TWI518551B - An electronic device controlling system and method thereof - Google Patents

An electronic device controlling system and method thereof Download PDF

Info

Publication number
TWI518551B
TWI518551B TW104103131A TW104103131A TWI518551B TW I518551 B TWI518551 B TW I518551B TW 104103131 A TW104103131 A TW 104103131A TW 104103131 A TW104103131 A TW 104103131A TW I518551 B TWI518551 B TW I518551B
Authority
TW
Taiwan
Prior art keywords
action
image
motion
command
instruction
Prior art date
Application number
TW104103131A
Other languages
Chinese (zh)
Other versions
TW201627817A (en
Inventor
李彥廷
洪西進
Original Assignee
國立臺灣科技大學
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 國立臺灣科技大學 filed Critical 國立臺灣科技大學
Priority to TW104103131A priority Critical patent/TWI518551B/en
Application granted granted Critical
Publication of TWI518551B publication Critical patent/TWI518551B/en
Publication of TW201627817A publication Critical patent/TW201627817A/en

Links

Landscapes

  • User Interface Of Digital Computer (AREA)

Description

一種電子裝置控制系統與其方法 Electronic device control system and method thereof

本發明係關於一種電子裝置控制系統與其方法,更明確地說,本發明係關於一種利用使用者肢體動作,來控制電子裝置的系統與方法。 The present invention relates to an electronic device control system and method thereof, and more particularly to a system and method for controlling an electronic device using a user's limb motion.

身處生活日益進步的現在,科技已不再是只屬於專業人士的儀器,而是與人們生活息息相關的工具。故良好的人機互動一直是現今持續改善的課題,從利用一般的按鈕,到觸控,以及聲控,已經造成資訊爆炸性的革命,讓更多人可以利用科技改善自己的生活。而現在影像處理以及人機互動方面,更是近年來更是相當熱門的領域,習知技術包含的產品例如任天堂的Wii,其採用紅外線當作讀取鏡頭並且結合三軸加速來達到與用者肢體動作的判斷與辨識,其後更有Xbox360的Kinect和PlayStation3的MOVE等產品問世。而汽車產業、Smart TV和行動手機等,亦漸漸朝此概念前進且相繼推出相關的產品,並朝向平民化與人性化的方向發展。綜觀下來影像處理結合與機器的互動確實具有其發展的潛力以及研究的價值。 Nowadays, with the progress of life, technology is no longer an instrument only for professionals, but a tool that is closely related to people's lives. Therefore, good human-computer interaction has been the subject of continuous improvement. From the use of general buttons, to touch, and voice control, it has caused an explosive revolution in information, enabling more people to use technology to improve their lives. Nowadays, image processing and human-computer interaction are more popular in recent years. The products included in the technology include Nintendo's Wii, which uses infrared light as a reading lens and combines three-axis acceleration to reach the user. The judgment and identification of limb movements, followed by Xbox360's Kinect and PlayStation3's MOVE and other products. The automobile industry, Smart TV and mobile phones have gradually moved toward this concept and launched related products one after another, and are developing towards civilians and humanization. Looking at the combination of image processing and machine interaction does have its potential for development and the value of research.

然而習知技術的人機介面產品,都需要一架專門主機並搭配周邊配備,主機與週邊配備的價格仍算昂貴,且用途上僅用作娛樂用。泛用性並不高。 However, the human-machine interface products of the prior art require a dedicated host and are equipped with peripheral equipment. The price of the host and the peripheral equipment is still expensive, and the use is only for entertainment. Generality is not high.

因應前述問題,本發明提出一種電子裝置控制系統,將人機介面建置在一ARM(Advanced RISC Machine)平台的嵌入式系統中,藉由辨識一使用者之一第一動作影像,將使用者的肢體動作影像作為輸入指令,來控制並操作電子裝置。其包含:一影像前處理模組、一影像辨識模組以及一影像比對模組。 In response to the foregoing problems, the present invention provides an electronic device control system for constructing a human-machine interface in an embedded system of an ARM (Advanced RISC Machine) platform, by identifying a user's first motion image and placing the user The limb motion image is used as an input command to control and operate the electronic device. The method comprises: an image pre-processing module, an image recognition module and an image comparison module.

影像前處理模組用於前處理第一動作影像以產生一第二動作影像;影像辨識模組則電性連接影像前處理模組,用以定義第二動作影像中之一動作部位,並且追蹤及辨識動作部位所對應之一動作,再輸出動作所形成的一動作指令;以及影像比對模組係電性連接影像辨識模組,影像比對模組包含有一動作指令資料庫,動作指令資料庫儲存有一預設動作指令,影像比對模組用以比對動作指令是否符合預設動作指令,若比對結果符合,則輸出對應預設動作指令之一控制指令以控制電子裝置。 The image pre-processing module is configured to pre-process the first motion image to generate a second motion image; the image recognition module is electrically connected to the image pre-processing module to define an action part of the second motion image, and track And identifying an action corresponding to the action part, and then outputting an action instruction formed by the action; and the image comparison module is electrically connected to the image recognition module, and the image comparison module includes an action instruction database and an action instruction data The library stores a preset action command, and the image comparison module is configured to compare whether the action command conforms to the preset action command. If the comparison result is consistent, outputting one of the preset action command control commands to control the electronic device.

其中動作部位係為使用者肢體的任一可動作部位,例如手、腳、眼或唇等;動作包含一手掌部移動軌跡以及一手掌部手勢,以及動作指令包含一手掌部移動軌跡指令以及一手掌部手勢指令。 The action part is any actionable part of the user's limb, such as the hand, the foot, the eye or the lip; the action includes a palm movement track and a palm gesture, and the action command includes a palm movement track instruction and a Hand gestures.

本發明亦提供一種電子裝置控制方法,其包含以下步驟:S1:接收一使用者之一第一動作影像;S2:前處理第一動作影像以產生一第二動作影像;S3:於第二動作影像中定義出一動作部位,並辨識動作部位所對應之一動作,再輸出動作所形成的一動作指令;以及S4:將動作指令與一預設動作指令進行比對;其中,若比對結果符合,則輸出對應該預設動作指令之一控制指令。 The present invention also provides an electronic device control method, comprising the steps of: S1: receiving a first motion image of a user; S2: pre-processing the first motion image to generate a second motion image; and S3: An action part is defined in the image, and one action corresponding to the action part is recognized, and then an action instruction formed by the action is output; and S4: the action instruction is compared with a preset action instruction; wherein, if the result is compared If it matches, the output controls the instruction corresponding to one of the preset action instructions.

其中動作部位係為使用者肢體的任一可動作部位,例如手、腳、眼或唇等;動作包含一手掌部移動軌跡以及一手掌部手勢,以及動作指令包含一手掌部移動軌跡指令以及一手掌部手勢指令。 The action part is any actionable part of the user's limb, such as the hand, the foot, the eye or the lip; the action includes a palm movement track and a palm gesture, and the action command includes a palm movement track instruction and a Hand gestures.

相較於習知技術,本發明所提供的一種電子裝置控制系統與方法,因將人機介面建置在一ARM(Advanced RISC Machine)平台的嵌入式系統中,使得只要附有接收端的電子產品,皆可利用本發明而讓一使用者只需要遠距離利用手勢動作的方式即可代替一般遙控器或觸控的功能,並達到方便、省時與低成本的功效。 Compared with the prior art, the electronic device control system and method provided by the present invention are built into an embedded system of an ARM (Advanced RISC Machine) platform, so that the electronic product with the receiving end is attached. The invention can be used to replace the function of the general remote controller or the touch control by a user only by using the gesture action at a long distance, and achieve the convenience, time saving and low cost.

1‧‧‧電子裝置控制系統 1‧‧‧Electronic device control system

11‧‧‧影像擷取裝置 11‧‧‧Image capture device

12‧‧‧影像前處理模組 12‧‧‧Image pre-processing module

13‧‧‧指令輸出模組 13‧‧‧Command output module

14‧‧‧影像辨識模組 14‧‧‧Image recognition module

16‧‧‧影像比對模組 16‧‧‧Image comparison module

2‧‧‧電子裝置控制方法 2‧‧‧Electronic device control method

C1‧‧‧控制指令 C1‧‧‧Control Instructions

S1~S5‧‧‧步驟 S1~S5‧‧‧Steps

S21~S24、S31~S33‧‧‧子步驟 Substeps S21~S24, S31~S33‧‧

圖一繪製了根據本發明之一實施例之功能方塊圖。 Figure 1 depicts a functional block diagram in accordance with an embodiment of the present invention.

圖二繪製了根據本發明之一實施例之步驟S1~S4之流程圖。 Figure 2 depicts a flow chart of steps S1 - S4 in accordance with an embodiment of the present invention.

圖三繪製了根據本發明之一實施例之子步驟S21~S24之流程圖。 Figure 3 depicts a flow chart of sub-steps S21-S24 in accordance with an embodiment of the present invention.

圖四繪製了根據本發明之一實施例之時間差相減法示意圖。 Figure 4 is a schematic diagram showing a time difference subtraction method in accordance with an embodiment of the present invention.

圖五A至圖五B繪製了根據本發明之一實施例之侵蝕運算示意圖。 Figures 5A through 5B illustrate schematic diagrams of erosion operations in accordance with an embodiment of the present invention.

圖六A至圖六B繪製了根據本發明之一實施例之擴張運算示意圖。 6A to 6B are diagrams showing an expansion operation according to an embodiment of the present invention.

圖七A至圖七B繪製了根據本發明之一實施例之連通結構標示示意圖。 7A through 7B are schematic diagrams showing the communication structure according to an embodiment of the present invention.

圖八繪製了根據本發明之一實施例之子步驟S31~S33之流程圖。 Figure 8 depicts a flow chart of sub-steps S31-S33 in accordance with an embodiment of the present invention.

為使本發明能更清楚的被說明,請參照以下本發明詳細說明及其中所包括之實例,以更容易地理解本發明。 In order to make the invention more apparent, the following detailed description of the invention and the examples thereof are included to provide a better understanding of the invention.

本說明書僅對本發明之必要元件作出陳述,且僅係用於說明本發明其中之可能之實施例,然而說明書之記述應不侷限本發明所主張之技術本質的權利範圍。除非於說明書有明確地排除其可能,否則本發明並不侷限於特定方法、流程、功能或手段。亦應瞭解的是,目前所述僅係本發明可能之實施例,在本發明之實施或測試中,可使用與本說明書所述材料相類似或等效之任何方法、流程、功能或手段。 This description is only for the purpose of illustrating the essential elements of the invention, and is only intended to illustrate the possible embodiments of the invention, but the description of the specification should not limit the scope of the technical nature of the claimed invention. The present invention is not limited to the specific methods, procedures, functions, or means unless the scope of the invention is specifically excluded. It is also to be understood that the presently described embodiments are merely possible embodiments of the present invention, and any methods, procedures, functions or means similar or equivalent to those described herein may be employed in the practice or testing of the invention.

除非有另外定義,否則本說明書所用之所有技術及科學術語,皆具有與熟習本發明所屬技術者通常所瞭解的意義相同之意義。儘管在本發明之實施或測試中,可使用與本說明書所述方法及材料相類似或等效之任何方法及手段,但本說明書目前所述者僅係實例方法、流程及其相關資料。 Unless otherwise defined, all technical and scientific terms used in the specification have the same meaning meaning Although any methods and means similar or equivalent to those described in the present specification can be used in the practice or testing of the present invention, the presently described embodiments are merely exemplary methods, procedures, and related materials.

再者,本說明書中所提及之一數目以上或以下,係包含數目本身。且應瞭解的是,本說明書揭示執行所揭示功能之某些方法、流程,均存在多種可執行相同功能之與所揭示結構有關之結構,且上述之結構通常均可達成相同結果。 Furthermore, one or more of the numbers mentioned in the specification include the number itself. It should be understood that the present disclosure discloses certain methods and processes for performing the disclosed functions. There are a variety of structures related to the disclosed structures that perform the same functions, and the above structures generally achieve the same result.

首先,請參閱圖一,圖一繪製了根據本發明之一實施例之功能方塊圖。本發明提供一種電子裝置控制系統1,將人機介面建置在一ARM(Advanced RISC Machine)平台的嵌入式系統中,藉由辨識一使用者之一第一動作影像,將使用者的肢體動作影像作為輸入指令,來控制並操作 電子裝置。其包含:一影像擷取裝置11、一影像前處理模組12、一指令輸出模組13、一影像辨識模組14以及一影像比對模組16。 First, referring to FIG. 1, FIG. 1 depicts a functional block diagram in accordance with an embodiment of the present invention. The present invention provides an electronic device control system 1 for constructing a human-machine interface in an embedded system of an ARM (Advanced RISC Machine) platform, and recognizing a user's limb motion by recognizing a first motion image of a user Image as an input command to control and operate Electronic device. The image capture device 11 includes an image capture device 12 , an image pre-processing module 12 , an instruction output module 13 , an image recognition module 14 , and an image comparison module 16 .

影像擷取裝置11電性連接影像前處理模組12,用以擷取使用 者之第一動作影像,並將第一動作影像輸出至影像前處理模組12;影像前處理模組12用於前處理第一動作影像以產生一第二動作影像;影像辨識模組14電性連接影像前處理模組12,用以定義第二動作影像中之一動作部位,並追蹤及辨識動作部位所對應之一動作,再輸出動作所形成的一動作指令;影像比對模組16電性連接影像辨識模組14,影像比對模組16包含有一動作指令資料庫,動作指令資料庫儲存有一預設動作指令,影像比對模組16用以比對動作指令是否符合預設動作指令,若比對結果符合,則輸出對應預設動作指令之一控制指令C1以控制電子裝置;指令輸出模組13電性連接影像比對模組16,用以傳輸控制指令C1至電子裝置來進行電子裝置的控制。 The image capturing device 11 is electrically connected to the image pre-processing module 12 for use in capturing The first motion image is output to the image pre-processing module 12; the image pre-processing module 12 is configured to pre-process the first motion image to generate a second motion image; the image recognition module 14 The image processing pre-processing module 12 is configured to define an action part of the second motion image, and track and recognize one action corresponding to the action part, and then output an action instruction formed by the action; the image comparison module 16 The image matching module 14 is electrically connected to the image matching module 16, and the image comparison module 16 includes an action instruction database. The action command database stores a preset action command, and the image comparison module 16 is configured to compare whether the action command conforms to the preset action. The instruction outputs a control command C1 corresponding to one of the preset action commands to control the electronic device; the command output module 13 is electrically connected to the image comparison module 16 for transmitting the control command C1 to the electronic device. Control of the electronic device.

其中,影像擷取裝置11得為一攝影機;動作部位係為該使用 者肢體的任一可動作部位,例如手、腳、眼或唇等。而動作部位所對應之動作包含一手掌部移動軌跡以及一手掌部手勢,以及動作所形成的動作指令則包含一手掌部移動軌跡指令以及一手掌部手勢指令。 Wherein, the image capturing device 11 is a camera; the action portion is the use Any actionable part of the limb, such as the hands, feet, eyes or lips. The action corresponding to the action part includes a palm movement track and a palm gesture, and the action command formed by the action includes a palm movement track command and a palm gesture command.

接著請參閱圖二,圖二繪製了根據本發明之一實施例之步驟 S1~S4流程圖。本發明亦提供一種電子裝置控制方法2,其包含以下步驟:S1:接收一使用者之一第一動作影像;S2:前處理第一動作影像以產生一第二動作影像;S3:於第二動作影像中定義出一動作部位,並辨識動作部位所對應之一動作,再輸出動作所形成的一動作指令;以及S4:將動作指 令與一預設動作指令進行比對;其中,若比對結果符合,則輸出對應該預設動作指令之一控制指令。 Referring next to FIG. 2, FIG. 2 depicts steps in accordance with an embodiment of the present invention. S1~S4 flow chart. The present invention also provides an electronic device control method 2, which includes the following steps: S1: receiving a first motion image of a user; S2: pre-processing the first motion image to generate a second motion image; S3: The action image defines an action part, and identifies one action corresponding to the action part, and then outputs an action instruction formed by the action; and S4: the action finger Aligning with a preset action instruction; wherein, if the comparison result is met, the output corresponds to one of the preset action instructions to control the instruction.

其中,步驟S1係利用影像擷取裝置11擷取使用者之第一動作 影像,而第一動作影像被傳輸至影像前處理模組12以進行步驟S2:前處理第一動作影像以產生一第二動作影像。 Step S1 uses the image capturing device 11 to capture the first action of the user. The image is transmitted to the image pre-processing module 12 to perform step S2: pre-processing the first motion image to generate a second motion image.

以下將以手部為動作部位為實施例,說明如何對第一動作影 像進行前處理,以產生一第二動作影像之流程。請參閱圖三,圖三繪製了根據本發明之一實施例之子步驟S21~S24流程圖。其中,步驟S2包含子步驟S21~S24,分別是S21:將第一動作影像中靜態背景去除,並留下至少一動作物體影像;S22:去除至少一動作物體影像中的雜訊;S23:加強至少一動作物體影像;以及S24:將至少一動作物體影像進行編號,以形成並輸出包含至少一動作物體影像之第二動作影像。 In the following, the hand as the action part is taken as an example to illustrate how to act on the first action. Like the pre-processing to generate a second motion image. Referring to FIG. 3, FIG. 3 depicts a flowchart of sub-steps S21-S24 according to an embodiment of the present invention. Step S2 includes sub-steps S21-S24, which are respectively S21: removing the static background in the first motion image and leaving at least one motion object image; S22: removing noise in at least one motion object image; S23: strengthening At least one motion object image; and S24: numbering at least one motion object image to form and output a second motion image including at least one motion object image.

請參閱圖四,圖四A繪製了根據本發明之一實施例之時間差 相減法示意圖。由於影像擷取裝置11所擷取的第一動作影像包含有一靜態背景影像以及使用者的動作物體動態影像,在子步驟S21中利用時間差相減法(Temporal Difference)來去除第一動作影像中的靜態背景,以留下至少一動作物體影像。時間差相減法(Temporal Difference)係將第一動作影像拆分成複數個影格(Frame),再將目前的影格與前一張影格進行相減,並將所得到的值做二值化,來將第一動作影像中至少一動作物體影像攝取出來,達成去除靜態背景之效果(如圖四所示)。接著進行子步驟S22:去除至少一動作物體影像中的雜訊。 Referring to FIG. 4, FIG. 4A depicts a time difference according to an embodiment of the present invention. Schematic diagram of the subtraction method. The first motion image captured by the image capturing device 11 includes a static background image and a motion image of the user's motion object. In sub-step S21, the Temporal Difference is used to remove the static in the first motion image. Background to leave at least one moving object image. The Temporal Difference method splits the first motion image into a plurality of frames, then subtracts the current frame from the previous frame, and binarizes the obtained values. At least one moving object image is captured in the first motion image to achieve the effect of removing the static background (as shown in FIG. 4). Sub-step S22 is then performed: removing noise in the image of at least one moving object.

請參閱圖五A至圖五B,圖五A至圖五B繪製了根據本發明之 一實施例之侵蝕運算示意圖。由於第一動作影像中的靜態背景是不會完全處於靜態的狀況,並且在光影變化下,時間差相減法往往會出現一些被當作移動的雜質或區塊等雜訊,如圖五A所示。子步驟S22中利用了侵蝕運算(Erosion)來去除至少一動作物體影像中的雜訊,如圖五B所示。 Please refer to FIG. 5A to FIG. 5B, and FIG. 5A to FIG. 5B are drawn according to the present invention. A schematic diagram of the erosion operation of an embodiment. Since the static background in the first motion image is not completely static, and under the light and shadow changes, the time difference subtraction method will often appear as noise or block noise as moving, as shown in Figure 5A. . In the sub-step S22, an erosion operation (Erosion) is used to remove noise in at least one moving object image, as shown in FIG. 5B.

請參閱圖六A至圖六B,圖六A至圖六B繪製了根據本發明之 一實施例之擴張運算示意圖。接著將以除去雜訊的至少一動作物體影像進行子步驟S23:加強至少一動作物體影像。因為至少一動作物體影像經過侵蝕運算後,除了去除雜訊外,也會讓至少一動作物體影像中感興趣的區塊看起來收縮,如圖六A所示。所以需要將這些區塊還原成侵蝕運算前的大小,如圖六B所示。 Please refer to FIG. 6A to FIG. 6B, and FIG. 6A to FIG. 6B are drawn according to the present invention. A schematic diagram of the expansion operation of an embodiment. Next, sub-step S23 is performed to remove at least one moving object image of the noise to enhance at least one moving object image. Because at least one moving object image undergoes an erosion operation, in addition to removing noise, the block of interest in at least one moving object image appears to shrink, as shown in FIG. 6A. Therefore, it is necessary to restore these blocks to the size before the erosion operation, as shown in Figure 6B.

接著將進行影像前處理的最後一子步驟S24:將該至少一動 作物體影像進行編號,以形成並輸出包含該至少一動作物體影像之該第二動作影像。請參閱圖七A至圖七B,圖七A至圖七B繪製了根據本發明之一實施例之連通結構標示示意圖。 Next, the last sub-step S24 of the image pre-processing will be performed: the at least one motion The object images are numbered to form and output the second motion image including the image of the at least one motion object. Referring to FIG. 7A to FIG. 7B, FIG. 7A to FIG. 7B are diagrams showing the connection structure indication according to an embodiment of the present invention.

經過前面一連串的處理後可得如圖七A所示的至少一動作 物體影像,其中有三團以白色像素表示的區塊,皆係為動作物體影像,吾人需要將每個相連的白色像素區塊分別標示,以輸出完成影像前處理,並包含至少一動作物體影像之第二動作影像。而本發明將採用連通結構標示(Connected Component Labeling)來完成區塊標示。連通結構標示是將不同區塊的屬性分門別類,將互相連接在一起的區塊給予相同的編號,如圖七B的編號1至3所示,而這些被編號的動作物體影像即為肢體影像的候選。 After a series of previous processes, at least one action as shown in FIG. 7A can be obtained. Object image, in which three groups of blocks represented by white pixels are images of moving objects, we need to mark each connected white pixel block separately to output image pre-processing, and include at least one moving object image. The second motion image. The invention will use the Connected Component Labeling to complete the block labeling. The connectivity structure designates the attributes of different blocks into different categories, and the blocks that are connected to each other are given the same number, as shown in the numbers 1 to 3 of Figure 7B, and the images of the numbered action objects are limb images. Candidate.

接著請參閱圖八,圖八繪製了根據本發明之一實施例之子步 驟S31~S33之流程圖。取得了第二動作影像之後,接下來將進行步驟S3:於該第二動作影像中定義出一動作部位,並辨識該動作部位所對應之一動作,再輸出該動作所形成的一動作指令。其中步驟S3內包含有以下子步驟:S31:將該至少一動作物體影像編號中非肢體影像的編號剔除,再進一步從該至少一動作物體影像中定義出該動作部位;S32:追蹤該動作部位;以及S33:辨識該動作部位所形成之一動作,並輸出該動作所形成的一動作指令。 Referring next to FIG. 8, FIG. 8 depicts substeps in accordance with an embodiment of the present invention. Steps S31~S33. After the second motion image is acquired, step S3 is performed: an action portion is defined in the second motion image, and one motion corresponding to the motion portion is recognized, and an action command formed by the motion is output. The step S3 includes the following sub-steps: S31: removing the number of the non-limb image in the image number of the at least one action object, and further defining the action part from the image of the at least one action object; S32: tracking the action part And S33: identifying an action formed by the action portion, and outputting an action command formed by the action.

在步驟S31中,本發明先後利用了相似比率法,來將第二動作影像裡至少一動作物體影像編號中不是使用者肢體影像的區塊剔除,亦即剔除圖七B中編號2與3的區塊,並利用Harris偵測法進一步定義出使用者的動作部位,亦即圖七B中的編號1。致此,已完成了動作部位的定義。其中此動作部位為手部或手掌部 In the step S31, the present invention successively utilizes the similarity ratio method to remove the blocks of the at least one action object image number in the second motion image that are not the user's limb image, that is, the numbers 2 and 3 in FIG. 7B are eliminated. Block, and use Harris detection to further define the user's action location, which is number 1 in Figure 7B. To this end, the definition of the action site has been completed. Where the action part is the hand or the palm

接著再進行子步驟S32:追蹤該動作部位。本發明利用了方向梯度直方圖法(Histogram of Oriented Gradients,HOG)以及平均位移法(Mean Shift)進行動作部位的追蹤。例如第一動作影像中動作部位的手掌部正在進行一順時針繞行運動、或是其他的任意運動。而在本步驟中本發明則是在每一影格中追蹤動作部位的位置。其中係以HOG中梯度特徵為主,平均位移法中的灰度(Grayscale)特徵為輔來進行。 Sub-step S32 is then performed: the action portion is tracked. The invention utilizes the Histogram of Oriented Gradients (HOG) and the Mean Shift method to track the action parts. For example, the palm portion of the action portion in the first motion image is undergoing a clockwise bypass motion or any other motion. In the present step, the present invention tracks the position of the action portion in each frame. Among them, the gradient features in HOG are mainly used, and the grayscale feature in the average displacement method is supplemented.

最後再進行子步驟S33:辨識該動作部位所形成之一動作,並輸出該動作所形成的一動作指令。本發明藉由時間差相減法(Temporal Difference)以及掃描投影法進行動作部位所進行之動作的辨識。例如利用時間差相減法來辨識動作部位(手掌部)的動作形成之移動軌跡,或是辨識手掌部是否作出握拳的手勢動作;若是動作部位(手掌部)用手指做出了 比數字的手勢動作,本發明則利用掃描投影法來辨識動作部位(手掌部)所比出的數字。最後再輸出包含了動作部位移動軌跡指令以及動作部位手勢指令的動作指令。 Finally, sub-step S33 is performed: identifying an action formed by the action part, and outputting an action command formed by the action. The present invention recognizes the action performed by the action site by the Temporal Difference method and the scan projection method. For example, the time difference subtraction method is used to identify the movement trajectory formed by the movement of the action part (the palm part), or whether the palm part is made to make a gesture of making a fist; if the action part (palm part) is made with a finger Compared with the digital gesture action, the present invention uses the scanning projection method to identify the number compared to the action portion (the palm portion). Finally, an operation command including an action part movement track command and an action part gesture command is output.

經過上述子步驟S31~S33,並輸出動作指令之後,最後再進 行步驟S4:將該動作指令與一預設動作指令進行比對。首先,本發明具有動作指令資料庫以預先記錄有預設動作指令,每一預設動作指令皆有一相對應的控制指令,例如當動作指令中手勢指令為握拳時,所對應的控制指令是關閉電子裝置;當動作指令中有動作部位移動軌跡指令時,則對應的控制指令是根據動作部位移動軌跡指令移動游標等。於本步驟中,將由步驟S3所輸出的動作指令之內容與預設動作指令進行比對,若動作指令內容相符於預設動作指令,則輸出預設動作指令所對應的控制指令來控制電子裝置。 After the above sub-steps S31-S33, and output the action command, and finally re-enter Step S4: The action instruction is compared with a preset action command. First, the present invention has an action instruction database to pre-record a preset action command, and each preset action command has a corresponding control command. For example, when the gesture command in the action command is a fist, the corresponding control command is closed. The electronic device; when there is an action part moving track command in the action command, the corresponding control command is to move the cursor according to the action part moving track command. In this step, the content of the action command outputted in step S3 is compared with the preset action command. If the action command content matches the preset action command, the control command corresponding to the preset action command is output to control the electronic device. .

綜上所言,本發明提供一種電子裝置控制系統與方法,控制 系統包含一影像前處理模組、一影像辨識模組以及一影像比對模組;控制方法包含步驟:S1:接收一使用者之一第一動作影像;S2:前處理第一動作影像以產生一第二動作影像;S3:於第二動作影像中定義出一動作部位,並辨識動作部位所對應之一動作,再輸出動作所形成的一動作指令;以及S4:將動作指令與一預設動作指令進行比對。本發明在接收第一動作影像之後由影像前處理模組執行步驟S2、再由影像辨識模組執行步驟S2~S3、最後由影像比對模組執行步驟S4。讓一使用者得以藉由本發明以肢體動作與手勢動作控制電子裝置。 In summary, the present invention provides an electronic device control system and method for controlling The system comprises an image pre-processing module, an image recognition module and an image comparison module; the control method comprises the steps of: S1: receiving a first motion image of a user; and S2: processing the first motion image to generate a second motion image; S3: defining an action part in the second action image, and identifying one action corresponding to the action part, and then outputting an action instruction formed by the action; and S4: setting the action instruction with a preset The motion instructions are compared. After receiving the first motion image, the image execution module performs step S2, and then the image recognition module performs steps S2 to S3, and finally the image comparison module executes step S4. Allowing a user to control the electronic device by limb motion and gesture action by the present invention.

相較於習知技術,本發明所提供的一種電子裝置控制系統與 方法,因將人機介面建置在一ARM(Advanced RISC Machine)平台的嵌入式系統中,使得只要附有接收端的電子產品,皆可利用本發明而讓一使用者只需要遠距離利用手勢動作的方式即可代替一般遙控器或觸控的功能,並達到方便、省時與低成本的功效。 Compared with the prior art, the present invention provides an electronic device control system and The method, because the human-machine interface is built in an embedded system of an ARM (Advanced RISC Machine) platform, so that as long as the electronic product with the receiving end is attached, the user can use the present invention to allow a user to only use the gesture action at a long distance. The way to replace the general remote control or touch function, and to achieve convenient, time-saving and low-cost effects.

以上較佳具體實施例之詳述,俾利清楚描述本發明之特徵與精神,而並非以上述所揭露之較佳具體實施例來對本發明之範疇加以限制。雖然本發明已以實施方式揭露如上,然其並非用以限定本發明,任何熟習此技藝者,在不脫離本發明之精神和範圍內,當可作各種之更動與潤飾,而本發明之保護範圍當視後附之申請專利範圍所界定者為準。 The above description of the preferred embodiments of the present invention is intended to be illustrative of the invention and the scope of the invention. Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and it is to be understood that the invention may be modified and modified without departing from the spirit and scope of the invention. The scope is subject to the definition of the scope of the patent application attached.

1‧‧‧電子裝置控制系統 1‧‧‧Electronic device control system

11‧‧‧影像擷取裝置 11‧‧‧Image capture device

12‧‧‧影像前處理模組 12‧‧‧Image pre-processing module

13‧‧‧指令輸出模組 13‧‧‧Command output module

14‧‧‧影像辨識模組 14‧‧‧Image recognition module

16‧‧‧影像比對模組 16‧‧‧Image comparison module

Claims (12)

一種電子裝置控制系統,藉由辨識一使用者之一第一動作影像,以操作一電子裝置;其包含:一影像前處理模組,用於前處理該第一動作影像以產生一第二動作影像;一影像辨識模組,電性連接該影像前處理模組,用以定義該第二動作影像中之一動作部位,並追蹤及辨識該動作部位所對應之一動作,再輸出該動作所形成的一動作指令;以及一影像比對模組,電性連接該影像辨識模組,該影像比對模組包含有一動作指令資料庫,該動作指令資料庫儲存有一預設動作指令,該影像比對模組用以比對該動作指令是否符合該預設動作指令,若比對結果符合,則輸出對應該預設動作指令之一控制指令以控制該電子裝置。 An electronic device control system for operating an electronic device by recognizing a first motion image of a user; comprising: an image pre-processing module, configured to pre-process the first motion image to generate a second motion An image recognition module electrically connected to the image pre-processing module for defining an action part of the second action image, and tracking and recognizing one of the actions corresponding to the action part, and then outputting the action Forming an action command; and an image comparison module electrically connected to the image recognition module, the image comparison module includes an action instruction database, wherein the action instruction database stores a preset action command, the image The comparison module is configured to compare whether the action instruction conforms to the preset action instruction, and if the comparison result is consistent, outputting a control instruction corresponding to one of the preset action instructions to control the electronic device. 如申請專利範圍第1項所述的電子控制裝置系統,其中該動作部位係為該使用者肢體的任一可動作部位,例如手、腳、眼或唇等。 The electronic control device system of claim 1, wherein the action portion is any actionable part of the user's limb, such as a hand, a foot, an eye or a lip. 如申請專利範圍第1項所述的電子控制裝置系統,其中該動作包含一手掌部移動軌跡以及一手掌部手勢,以及該動作指令包含一手掌部移動軌跡指令以及一手掌部手勢指令。 The electronic control device system of claim 1, wherein the action comprises a palm movement track and a palm gesture, and the action command comprises a palm movement track command and a palm gesture command. 如申請專利範圍第1項所述的電子控制裝置系統,其進一步包含有一影像擷取裝置,電性連接該影像前處理模組,用以擷取該使用者之該第一動作影像,並輸出至該影像前處理模組。 The electronic control device system of claim 1, further comprising an image capture device electrically connected to the image pre-processing module for capturing the first motion image of the user and outputting To the image pre-processing module. 如申請專利範圍第1項所述的電子控制裝置系統,其進一步包含一指令 輸出模組,電性連接該影像比對模組,用以傳輸該控制指令至該電子裝置來進行該電子裝置的控制。 The electronic control device system of claim 1, further comprising an instruction The output module is electrically connected to the image comparison module for transmitting the control command to the electronic device for controlling the electronic device. 一種電子裝置控制方法,其包含以下步驟:S1:接收一使用者之一第一動作影像;S2:前處理該第一動作影像以產生一第二動作影像;S3:於該第二動作影像中定義出一動作部位,並辨識該動作部位所對應之一動作,再輸出該動作所形成的一動作指令;以及S4:將該動作指令與一預設動作指令進行比對;其中,若比對結果符合,則輸出對應該預設動作指令之一控制指令。 An electronic device control method includes the following steps: S1: receiving a first motion image of a user; S2: pre-processing the first motion image to generate a second motion image; and S3: in the second motion image Defining an action part, and identifying one action corresponding to the action part, and then outputting an action instruction formed by the action; and S4: comparing the action instruction with a preset action instruction; wherein, if the comparison If the result is met, the output controls the instruction corresponding to one of the preset action instructions. 如申請專利範圍第6項所述的電子控制裝置方法,其中步驟S2更包含以下子步驟:S21:將該第一動作影像中靜態背景去除,並留下至少一動作物體影像;S22:去除該至少一動作物體影像中的雜訊;S23:加強該至少一動作物體影像;以及S24:將該至少一動作物體影像進行編號,以形成並輸出包含該至少一動作物體影像之該第二動作影像。 The electronic control device method of claim 6, wherein the step S2 further comprises the following substeps: S21: removing the static background in the first motion image and leaving at least one motion object image; S22: removing the At least one motion object image noise; S23: enhancing the at least one motion object image; and S24: numbering the at least one motion object image to form and output the second motion image including the at least one motion object image . 如申請專利範圍第7項所述的電子控制裝置方法,其中進行步驟S21的方法包含時間差相減法(Temporal Difference);進行步驟S22的方法包含侵蝕運算(Erosion);進行步驟S23的方法包含擴張運算(Dilation);進行步驟S24的方法包含連通結構標示(Connected Component Labeling)。 The electronic control device method according to claim 7, wherein the method of performing step S21 includes a temporal difference subtraction method; the method of performing step S22 includes an erosion operation (Erosion); and the method of performing step S23 includes an expansion operation. (Dilation); The method of performing step S24 includes Connected Component Labeling. 如申請專利範圍第7項所述的電子控制裝置方法,其中步驟S3更包含以 下子步驟:S31:將該至少一動作物體影像編號中非肢體影像的編號剔除,再進一步從該至少一動作物體影像中定義出該動作部位;S32:追蹤該動作部位;以及S33:辨識該動作部位所形成之一動作,並輸出該動作所形成的一動作指令。 The electronic control device method of claim 7, wherein the step S3 further comprises Sub-step: S31: the number of the non-limb image in the image number of the at least one action object is removed, and the action part is further defined from the at least one action object image; S32: tracking the action part; and S33: identifying the action One of the parts forms an action and outputs an action command formed by the action. 如申請專利範圍第9項所述的電子控制裝置方法,其中進行步驟S31的方法包含相似比率法以及Harris偵測法;進行步驟S32的方法包含方向梯度直方圖法(Histogram of Oriented Gradients,HOG)以及平均位移法(Mean Shift);進行步驟S33的方法包含時間差相減法(Temporal Difference)以及掃描投影法。 The electronic control device method of claim 9, wherein the method of performing step S31 comprises a similar ratio method and a Harris detection method; and the method of performing step S32 comprises a Histogram of Oriented Gradients (HOG). And an average displacement method (Mean Shift); the method of performing step S33 includes a temporal difference subtraction method (Temporal Difference) and a scanning projection method. 如申請專利範圍第6項所述的電子控制裝置方法,其中該動作部位係為該使用者肢體的任一可動作部位,例如手、腳、眼或唇等。 The electronic control device method according to claim 6, wherein the action portion is any actionable part of the user's limb, such as a hand, a foot, an eye or a lip. 如申請專利範圍第6項所述的電子控制裝置方法,其中該動作包含一手掌部移動軌跡以及一手掌部手勢,以及該動作指令包含一手掌部移動軌跡指令以及一手掌部手勢指令。 The electronic control device method of claim 6, wherein the action comprises a palm movement track and a palm gesture, and the action command comprises a palm movement track command and a palm gesture command.
TW104103131A 2015-01-30 2015-01-30 An electronic device controlling system and method thereof TWI518551B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
TW104103131A TWI518551B (en) 2015-01-30 2015-01-30 An electronic device controlling system and method thereof

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
TW104103131A TWI518551B (en) 2015-01-30 2015-01-30 An electronic device controlling system and method thereof

Publications (2)

Publication Number Publication Date
TWI518551B true TWI518551B (en) 2016-01-21
TW201627817A TW201627817A (en) 2016-08-01

Family

ID=55640443

Family Applications (1)

Application Number Title Priority Date Filing Date
TW104103131A TWI518551B (en) 2015-01-30 2015-01-30 An electronic device controlling system and method thereof

Country Status (1)

Country Link
TW (1) TWI518551B (en)

Also Published As

Publication number Publication date
TW201627817A (en) 2016-08-01

Similar Documents

Publication Publication Date Title
JP6323040B2 (en) Image processing apparatus, image processing method, and program
US8339359B2 (en) Method and system for operating electric apparatus
US8644556B2 (en) Image processing apparatus and method and program
US8897490B2 (en) Vision-based user interface and related method
TWI499966B (en) Interactive operation method of electronic apparatus
US8938124B2 (en) Computer vision based tracking of a hand
KR101929077B1 (en) Image identificaiton method and image identification device
JP2007316882A (en) Remotely-controlled device and method
Wu et al. Robust fingertip detection in a complex environment
US20160334877A1 (en) Real-time 3d gesture recognition and tracking system for mobile devices
JP2020067999A (en) Method of virtual user interface interaction based on gesture recognition and related device
US20160086349A1 (en) Tracking hand pose using forearm-hand model
US20230195301A1 (en) Text input method and apparatus based on virtual keyboard
KR100562632B1 (en) A video based handwriting recognition system and method
Sharma et al. Air-swipe gesture recognition using OpenCV in Android devices
EP2786312A1 (en) A gesture recognition method, an apparatus and a computer program for the same
Dulayatrakul et al. Robust implementation of hand gesture recognition for remote human-machine interaction
US9317136B2 (en) Image-based object tracking system and image-based object tracking method
Choi et al. RGB-D camera-based hand shape recognition for human-robot interaction
Bakar et al. Real-time rotation invariant hand tracking using 3D data
Singh Recognizing hand gestures for human computer interaction
TWI518551B (en) An electronic device controlling system and method thereof
Choondal et al. Design and implementation of a natural user interface using hand gesture recognition method
Siam et al. Human computer interaction using marker based hand gesture recognition
KR102107182B1 (en) Hand Gesture Recognition System and Method

Legal Events

Date Code Title Description
MM4A Annulment or lapse of patent due to non-payment of fees