TWI648556B - Slam and gesture recognition method - Google Patents
Slam and gesture recognition method Download PDFInfo
- Publication number
- TWI648556B TWI648556B TW107107503A TW107107503A TWI648556B TW I648556 B TWI648556 B TW I648556B TW 107107503 A TW107107503 A TW 107107503A TW 107107503 A TW107107503 A TW 107107503A TW I648556 B TWI648556 B TW I648556B
- Authority
- TW
- Taiwan
- Prior art keywords
- image
- gesture
- unit
- positioning
- map construction
- Prior art date
Links
Landscapes
- User Interface Of Digital Computer (AREA)
- Image Analysis (AREA)
Abstract
本案係關於一種同步定位與地圖建構及手勢辨識方法,包括步驟:提供頭戴式電子裝置;以影像擷取模組進行影像擷取,以取得擷取影像;影像處理單元對擷取影像進行前處理動作,產生顯示影像;複製顯示影像並分別傳送至動態影像過濾單元及手勢影像擷取單元;動態影像過濾單元過濾顯示影像,以除去顯示影像中之動態影像,進而取得空間影像;空間定位演算法單元根據空間影像進行同步定位與地圖建構;手勢影像擷取單元分析顯示影像,並擷取顯示影像中之手勢影像;以及細部手勢辨識演算法單元根據手勢影像進行細部手勢辨識。藉此可減少產品體積、重量與製造成本。The present invention relates to a method for synchronous positioning and map construction and gesture recognition, comprising the steps of: providing a head-mounted electronic device; capturing an image by using an image capturing module to obtain a captured image; and the image processing unit before capturing the image Processing the action to generate a display image; copying the display image and transmitting it to the motion image filtering unit and the gesture image capturing unit respectively; the motion image filtering unit filters the displayed image to remove the dynamic image in the displayed image, thereby obtaining a spatial image; spatial positioning calculation The method unit performs synchronous positioning and map construction according to the spatial image; the gesture image capturing unit analyzes and displays the image, and captures the gesture image in the displayed image; and the detailed gesture recognition algorithm unit performs detailed gesture recognition according to the gesture image. This reduces product size, weight and manufacturing costs.
Description
本案係關於一種影像之空間定位與手勢辨識方法,尤指一種應用於擴增實境之同步定位與地圖建構及手勢辨識方法。This case is about a method for spatial positioning and gesture recognition of images, especially a method for synchronous positioning, map construction and gesture recognition applied in augmented reality.
目前在擴增實境(Augmented Reality, AR)的應用上,為了能與虛擬物件互動,人機介面的研究日益蓬勃。由於穿戴穿戴式的擴增實境裝置或其他頭戴式電子裝置時,多半無法採用傳統的滑鼠控制或是觸控控制等方式,故多種型態的人機介面方式持續地被提出。At present, in the application of Augmented Reality (AR), in order to interact with virtual objects, the research on human-machine interface is growing vigorously. Since wearable augmented reality devices or other head-mounted electronic devices are often unable to adopt conventional mouse control or touch control, various types of human-machine interface methods are continuously proposed.
舉例而言,擴增實境可透過特製的控制器來與虛擬物件進行互動,其反應速度靈敏、控制方式多元,但仍需因應不同的使用環境採用多種額外的控制器,且需學習並記憶每一種控制器的操作方式,無法達到直覺使用的效果。因此,無須使用額外控制器的手勢控制方式遂發展成一主要的人機介面方式。For example, augmented reality can interact with virtual objects through a special controller. The response speed is sensitive and the control method is diversified. However, multiple additional controllers need to be used according to different usage environments, and need to learn and memorize. The way each controller operates can't achieve the effect of intuitive use. Therefore, it is not necessary to use the gesture control method of the additional controller to develop into a main human-machine interface mode.
然而,現行對於手勢控制方式的處理,仍具有許多缺點。請參閱第1圖及第2圖,其中第1圖係顯示習知頭戴式電子裝置之架構方塊圖,以及第2圖係顯示習知頭戴式電子裝置擷取目標手勢及目標場景並進行演算之示意圖。由第1圖及第2圖可得知,習知頭戴式電子裝置1主要包括控制單元10、第一影像擷取模組11、手勢影像處理器12、第二影像擷取模組13、場景影像處理器14、手勢辨識演算法單元15以及空間定位演算法單元16。其中,第一影像擷取模組11及第二影像擷取模組13在硬體上及功能上為二組彼此獨立的影像擷取模組,第一影像擷取模組11係對目標手勢進行影像擷取,並經由手勢影像處理器12處理後,將擷取到的目標手勢影像傳送至手勢辨識演算法單元15進行演算,以得到對應的手勢;第二影像擷取模組13係對目標場景進行影像擷取,並經由場景影像處理器14處理後,將擷取到的目標場景影像傳送至空間定位演算法單元16進行演算,以完成空間定位。然而,此架構必須配置二組獨立的影像擷取模組,且當欲配合其他功能時,可能需要因應需求再增添額外的影像擷取模組,使得體積及重量相對地提升,不僅製造的成本較高,也使得使用者的配戴體驗不佳。However, the current handling of gesture control methods still has a number of disadvantages. Please refer to FIG. 1 and FIG. 2 , wherein FIG. 1 is a block diagram showing the architecture of a conventional head-mounted electronic device, and FIG. 2 is a diagram showing a conventional head-mounted electronic device capturing target gestures and target scenes. Schematic diagram of the calculation. As can be seen from the first and second figures, the conventional head-mounted electronic device 1 mainly includes a control unit 10, a first image capturing module 11, a gesture image processor 12, and a second image capturing module 13. The scene image processor 14, the gesture recognition algorithm unit 15, and the spatial positioning algorithm unit 16. The first image capturing module 11 and the second image capturing module 13 are two independent image capturing modules on the hardware and the function, and the first image capturing module 11 is configured to target the target gesture. After the image capture is performed and processed by the gesture image processor 12, the captured target gesture image is transmitted to the gesture recognition algorithm unit 15 for calculation to obtain a corresponding gesture; the second image capture module 13 is paired The target scene is image captured, and processed by the scene image processor 14, the captured target scene image is transmitted to the spatial positioning algorithm unit 16 for calculation to complete spatial positioning. However, this architecture must be configured with two separate sets of image capture modules. When you want to work with other functions, you may need to add additional image capture modules to meet the needs, so that the volume and weight are relatively increased, not only the cost of manufacturing. Higher, also makes the user's wearing experience poor.
故此,如何發展一種採用單一影像擷取模組搭配適度的過濾,讓擷取影像可以同步使用於空間定位、地圖建構與手勢辨識之同步定位與地圖建構及手勢辨識方法,實為目前尚待解決的問題。Therefore, how to develop a single image capture module with moderate filtering, so that captured images can be used simultaneously for spatial positioning, map construction and gesture recognition synchronous positioning and map construction and gesture recognition methods, which is still to be solved The problem.
本案之主要目的為提供一種同步定位與地圖建構及手勢辨識方法,俾解決並改善前述先前技術之問題與缺點。The main purpose of the present invention is to provide a method for synchronous positioning, map construction and gesture recognition, which solves and improves the problems and disadvantages of the prior art described above.
本案之另一目的為提供一種同步定位與地圖建構及手勢辨識方法,採用單一的影像擷取模組擷取同時包括目標場景及目標手勢之一擷取影像,並透過動態影像過濾單元及手勢影像擷取單元等影像前處理單元針對各個演算法作前處理產生顯示影像,最後再由空間定位演算法單元以空間定位演算法進行同步定位與地圖建構,並由細部手勢辨識演算法單元以手勢辨識演算法進行細部手勢辨識,不僅可有效地減少產品體積與重量,同時增進使用者體驗並達成產品微小化的目的,更可進一步降低製造成本。Another object of the present invention is to provide a method for synchronous positioning, map construction and gesture recognition, which uses a single image capturing module to capture one of the target scene and the target gesture, and captures the image through the motion image filtering unit and the gesture image. The image pre-processing unit such as the capture unit performs pre-processing on each algorithm to generate a display image, and finally the spatial positioning algorithm unit performs spatial positioning algorithm for synchronous positioning and map construction, and the detailed gesture recognition algorithm unit uses gesture recognition. The algorithm performs detailed gesture recognition, which not only can effectively reduce the volume and weight of the product, but also enhance the user experience and achieve the purpose of miniaturization of products, and further reduce the manufacturing cost.
本案之另一目的為提供一種同步定位與地圖建構及手勢辨識方法,透過單一影像擷取模組擷取同時包括目標場景及目標手勢的全彩影像,其影像內容完整且可提供各類軟硬體進行影像處理及後續應用,具有良好的擴充性。Another object of the present invention is to provide a method for synchronous positioning, map construction and gesture recognition, which utilizes a single image capture module to capture a full-color image including both a target scene and a target gesture, and the image content is complete and can provide various types of soft and hard images. The image processing and subsequent application have good scalability.
為達上述目的,本案之一較佳實施態樣為提供一種同步定位與地圖建構及手勢辨識方法,包括步驟:(a)提供一頭戴式電子裝置;(b)以該頭戴式電子裝置之一影像擷取模組進行影像擷取,以取得一擷取影像;(c)以該頭戴式電子裝置之一影像處理單元對該擷取影像進行一前處理動作,產生一顯示影像;(d)複製該顯示影像並分別傳送至該頭戴式電子裝置之一動態影像過濾單元及一手勢影像擷取單元;(e)該動態影像過濾單元過濾該顯示影像,以除去該顯示影像中之動態影像,進而取得一空間影像;(f)該頭戴式電子裝置之一空間定位演算法單元根據該空間影像進行同步定位與地圖建構;(g)該手勢影像擷取單元分析該顯示影像,並擷取該顯示影像中之一手勢影像;以及(h)該頭戴式電子裝置之一細部手勢辨識演算法單元根據該手勢影像進行細部手勢辨識。In order to achieve the above object, a preferred embodiment of the present invention provides a method for synchronous positioning and map construction and gesture recognition, comprising the steps of: (a) providing a head mounted electronic device; and (b) using the head mounted electronic device An image capturing module performs image capturing to obtain a captured image; and (c) performing a pre-processing operation on the captured image by an image processing unit of the head mounted electronic device to generate a display image; (d) copying the display image and transmitting it to one of the dynamic image filtering unit of the head mounted electronic device and a gesture image capturing unit; (e) the dynamic image filtering unit filters the display image to remove the displayed image a moving image to obtain a spatial image; (f) a spatial positioning algorithm unit of the head mounted electronic device performs synchronous positioning and map construction according to the spatial image; (g) the gesture image capturing unit analyzes the displayed image And capturing a gesture image of the display image; and (h) the detail gesture recognition algorithm unit of the head mounted electronic device performs detailed gesture recognition according to the gesture image.
體現本案特徵與優點的一些典型實施例將在後段的說明中詳細敘述。應理解的是本案能夠在不同的態樣上具有各種的變化,其皆不脫離本案的範圍,且其中的說明及圖示在本質上係當作說明之用,而非架構於限制本案。Some exemplary embodiments embodying the features and advantages of the present invention are described in detail in the following description. It is to be understood that the present invention is capable of various modifications in various aspects, and is not to be construed as a limitation.
請參閱第3圖、第4圖及第5圖,其中第3圖係顯示本案較佳實施例之同步定位與地圖建構及手勢辨識方法之流程圖,第4圖係顯示適用本案同步定位與地圖建構及手勢辨識方法之頭戴式電子裝置之架構方塊圖,以及第5圖係顯示以第4圖所示之頭戴式電子裝置擷取擷取影像並進行演算之示意圖。如第3圖、第4圖及第5圖所示,適用於本案同步定位與地圖建構及手勢辨識方法之頭戴式電子裝置係至少包括中央處理單元20、影像擷取模組21、影像處理單元22、動態影像過濾單元23、空間定位演算法單元24、手勢影像擷取單元25以及細部手勢辨識演算法單元26。其中,中央處理單元20係架構於控制頭戴式電子裝置2之運作。影像擷取模組21係用以擷取擷取影像。影像處理單元22係與中央處理單元20相連接。動態影像過濾單元23係與中央處理單元20相連接,且空間定位演算法單元24係與該動態影像過濾單元23相連接。手勢影像擷取單元25係與中央處理單元20相連接,且細部手勢辨識演算法單元26係與手勢影像擷取單元25相連接。Please refer to FIG. 3, FIG. 4 and FIG. 5 , wherein FIG. 3 is a flow chart showing the synchronous positioning, map construction and gesture recognition method of the preferred embodiment of the present invention, and FIG. 4 is a view showing the synchronous positioning and map applicable to the present case. The block diagram of the head-mounted electronic device of the construction and gesture recognition method, and the fifth figure show the schematic diagram of capturing the image and performing calculation by the head-mounted electronic device shown in FIG. As shown in FIG. 3, FIG. 4 and FIG. 5, the head mounted electronic device suitable for the synchronous positioning, map construction and gesture recognition method of the present invention includes at least a central processing unit 20, an image capturing module 21, and image processing. The unit 22, the motion image filtering unit 23, the spatial positioning algorithm unit 24, the gesture image capturing unit 25, and the detailed gesture recognition algorithm unit 26. The central processing unit 20 is configured to control the operation of the head mounted electronic device 2. The image capturing module 21 is configured to capture captured images. The image processing unit 22 is connected to the central processing unit 20. The motion picture filtering unit 23 is connected to the central processing unit 20, and the spatial positioning algorithm unit 24 is connected to the motion picture filtering unit 23. The gesture image capturing unit 25 is connected to the central processing unit 20, and the detailed gesture recognition algorithm unit 26 is connected to the gesture image capturing unit 25.
本案較佳實施例之同步定位與地圖建構及手勢辨識方法係包括如下步驟。首先,如步驟S100所示,提供頭戴式電子裝置2。其次,如步驟S200所示,以頭戴式電子裝置2之影像擷取模組21進行影像擷取,以取得一擷取影像。接著,如步驟S300所示,以頭戴式電子裝置2之影像處理單元22對擷取影像進行前處理動作,以產生一顯示影像,該前處理動作包括但不限於鏡頭校正(Lens Shading Correction)、自動獲得(AutoGain)以及自動曝光(AutoExposure)等。然後,如步驟S400所示,複製經過前處理後的顯示影像並分別傳送至頭戴式電子裝置2之動態影像過濾單元23及手勢影像擷取單元25,亦即動態影像過濾單元23及手勢影像擷取單元25於此步驟會接收到相同的經過前處理動作的顯示影像。在步驟S400完成後,步驟S500與步驟S700係可同時執行於步驟S400之後,亦可存在時間差地分別執行,步驟S600係執行於步驟S500之後,且步驟S800係執行於步驟S700之後。在步驟S500中,動態影像過濾單元23過濾顯示影像,以除去顯示影像中之動態影像,進而取得空間影像,換言之,為了進行準確的空間定位,應就顯示影像排除其中的動態影像例如移動中的手或其他移動中的物件。在步驟S600中,頭戴式電子裝置2之空間定位演算法單元24根據空間影像進行同步定位與地圖建構。在步驟S700中,手勢影像擷取單元25分析顯示影像,並擷取顯示影像中之手勢影像。在步驟S800中,頭戴式電子裝置2之細部手勢辨識演算法單元26根據手勢影像進行細部手勢辨識。The method for synchronous positioning, map construction and gesture recognition in the preferred embodiment of the present invention includes the following steps. First, as shown in step S100, the head mounted electronic device 2 is provided. Next, as shown in step S200, the image capturing module 21 of the head mounted electronic device 2 performs image capturing to obtain a captured image. Then, as shown in step S300, the image processing unit 22 of the head mounted electronic device 2 performs a pre-processing operation on the captured image to generate a display image, which includes, but is not limited to, lens correction (Lens Shading Correction). , AutoGain and Auto Exposure. Then, as shown in step S400, the pre-processed display image is copied and transmitted to the motion image filtering unit 23 and the gesture image capturing unit 25 of the head mounted electronic device 2, that is, the motion image filtering unit 23 and the gesture image. The capture unit 25 receives the same pre-processed display image at this step. After the step S400 is completed, the step S500 and the step S700 may be performed simultaneously after the step S400, or may be performed separately with the time difference, the step S600 is performed after the step S500, and the step S800 is performed after the step S700. In step S500, the motion image filtering unit 23 filters the display image to remove the motion image in the displayed image, thereby acquiring the spatial image. In other words, in order to perform accurate spatial positioning, the moving image should be excluded from the displayed image, for example, in motion. Hand or other moving objects. In step S600, the spatial positioning algorithm unit 24 of the head mounted electronic device 2 performs synchronous positioning and map construction based on the spatial image. In step S700, the gesture image capturing unit 25 analyzes the displayed image and captures the gesture image in the displayed image. In step S800, the detailed gesture recognition algorithm unit 26 of the head mounted electronic device 2 performs detailed gesture recognition based on the gesture image.
根據本案之構想,中央處理單元20可為中央處理器,且於一些實施例中,上述之步驟S400較佳係以中央處理單元20實現,但不以此為限。影像擷取模組21較佳為包括至少二個鏡頭及二個感光元件之單一影像擷取模組,由於空間計算若以三角函數為之,至少需具備二個已知相對位置關係的鏡頭與感光元件,同時考量使產品微小化並增進使用者體驗的取向上,本案中的影像擷取模組21最佳為採用二個鏡頭搭配二個感光元件,但不以此為限。影像處理單元22係可為但不限於一影像訊號處理器(Image Signal Processor, ISP)。此外,動態影像過濾單元23、手勢影像擷取單元25、空間定位演算法單元24及細部手勢辨識演算法單元26可皆為軟體單元,且透過中央處理單元20控制以實現執行,然亦不以此為限。According to the concept of the present invention, the central processing unit 20 can be a central processing unit. In some embodiments, the step S400 is preferably implemented by the central processing unit 20, but is not limited thereto. The image capturing module 21 preferably includes a single image capturing module including at least two lenses and two photosensitive elements. Since the space calculation is performed by a trigonometric function, at least two lenses with known relative positional relationships are required. The image capturing module 21 in the present invention preferably uses two lenses to match two photosensitive elements, but is not limited thereto. The image processing unit 22 can be, but not limited to, an Image Signal Processor (ISP). In addition, the motion image filtering unit 23, the gesture image capturing unit 25, the spatial positioning algorithm unit 24, and the detailed gesture recognition algorithm unit 26 may all be software units, and are controlled by the central processing unit 20 to implement, but not This is limited.
進一步地,本案之頭戴式電子裝置2可進一步包括記憶體單元27及微投影顯示單元28。其中,記憶體單元27係與中央處理單元20相連接,微投影顯示單元28係與中央處理單元20相連接。其中,至少前述之一手勢影像,例如於某一特定瞬時之手勢影像,係儲存於記憶體單元27,以用於與前一個或後一個手勢影像比對或其他相類似之用途,但不以此為限。頭戴式電子裝置2在執行本案之同步定位與地圖建構及手勢辨識方法後,係相應地顯示在微投影顯示單元28上。換言之,使用者最終係於微投影顯示單元28上觀看到互動的結果。Further, the head mounted electronic device 2 of the present invention may further include a memory unit 27 and a micro projection display unit 28. The memory unit 27 is connected to the central processing unit 20, and the micro-projection display unit 28 is connected to the central processing unit 20. Wherein at least one of the aforementioned gesture images, such as a gesture image at a particular moment, is stored in the memory unit 27 for comparison with the previous or next gesture image or other similar uses, but not This is limited. The head-mounted electronic device 2 is displayed on the micro-projection display unit 28 after performing the synchronous positioning, map construction and gesture recognition methods of the present invention. In other words, the user ultimately views the result of the interaction on the micro-projection display unit 28.
在動態影像過濾方面,請參閱第6圖並配合第3圖及第4圖,其中第6圖係顯示第3圖所示之步驟S500之細部流程圖。如第3圖、第4圖及第6圖所示,本案同步定位與地圖建構及手勢辨識方法之步驟S500係包括細部流程步驟如下。如步驟S520所示,動態影像過濾單元23將顯示影像轉換為第一灰階化影像。接著,如步驟S540所示,動態影像過濾單元23根據第一灰階化影像計算一深度資訊。然後,如步驟S560所示,動態影像過濾單元23透過加速穩健特徵(Speeded Up Robust Features, SURF)取得複數個特徵點,並將複數個特徵點製作成三維點雲(Point Cloud)。再來,如步驟S580所示,根據每一幀(Frame)對應之三維點雲過濾並移除移動物體產生的點雲資料,進而取得空間影像。In terms of motion picture filtering, please refer to FIG. 6 and cooperate with FIG. 3 and FIG. 4, wherein FIG. 6 shows a detailed flowchart of step S500 shown in FIG. As shown in FIG. 3, FIG. 4 and FIG. 6, the step S500 of the simultaneous positioning and map construction and gesture recognition method of the present case includes the detailed process steps as follows. As shown in step S520, the motion image filtering unit 23 converts the display image into the first grayscale image. Next, as shown in step S540, the motion picture filtering unit 23 calculates a depth information according to the first grayscale image. Then, as shown in step S560, the motion picture filtering unit 23 obtains a plurality of feature points through the Speeded Up Robust Features (SURF), and creates a plurality of feature points into a three-dimensional point cloud. Then, as shown in step S580, the point cloud data generated by the moving object is filtered according to the three-dimensional point cloud corresponding to each frame, and the spatial image is obtained.
經由動態影像過濾單元23完成前處理後,在空間定位演算法單元24所執行之步驟係說明如後。請參閱第7圖並配合第3圖及第4圖,其中第7圖係顯示第3圖所示之步驟S600之細部流程圖。如第3圖、第4圖及第7圖所示,本案同步定位與地圖建構及手勢辨識方法之步驟S600係包括細部流程步驟如下。如步驟S620所示,空間定位演算法單元24係以空間定位演算法追蹤複數個特徵點的主方向,添加特徵點向量以與前一個關鍵幀匹配計算影像擷取模組之相機(即鏡頭或感光元件)之初始位置,並在當前幀以及局部地圖間找尋匹配點,以最佳化當前幀的位置並決定次一個關鍵幀。接著,如步驟S640所示,判斷當前幀是否為新的關鍵幀。當步驟S640之判斷結果為是,於步驟S640之後係依序執行步驟S660及步驟S680。在步驟S660中,係將當前幀加入局部地圖並更新,建立新的關鍵幀與其他關鍵幀的連接關係,以及更新新的關鍵幀與其他關鍵幀的特徵點匹配關係,以生成三維地圖。另在步驟S680中,係取得空間資訊,並對該空間資訊最佳化,以完成同步定位與地圖建構。After the pre-processing is completed by the motion picture filtering unit 23, the steps performed by the spatial positioning algorithm unit 24 are as follows. Please refer to FIG. 7 and cooperate with FIG. 3 and FIG. 4, wherein FIG. 7 shows a detailed flow chart of step S600 shown in FIG. As shown in FIG. 3, FIG. 4 and FIG. 7, the step S600 of the synchronous positioning and map construction and gesture recognition method of the present case includes the detailed process steps as follows. As shown in step S620, the spatial positioning algorithm unit 24 tracks the main directions of the plurality of feature points by using the spatial positioning algorithm, and adds the feature point vector to match the previous key frame to calculate the camera of the image capturing module (ie, the lens or The initial position of the photosensitive element) and find a matching point between the current frame and the partial map to optimize the position of the current frame and determine the next key frame. Next, as shown in step S640, it is determined whether the current frame is a new key frame. When the determination result in step S640 is YES, step S660 and step S680 are sequentially executed after step S640. In step S660, the current frame is added to the local map and updated, the connection relationship between the new key frame and other key frames is established, and the feature point matching relationship between the new key frame and other key frames is updated to generate a three-dimensional map. In addition, in step S680, spatial information is obtained, and the spatial information is optimized to complete synchronization positioning and map construction.
在手勢分析方面,請參閱第8圖並配合第3圖及第4圖,其中第8圖係顯示第3圖所示之步驟S700之細部流程圖。如第3圖、第4圖及第8圖所示,本案同步定位與地圖建構及手勢辨識方法之步驟S700係包括細部流程步驟如下。如步驟S725所示,手勢影像擷取單元25將顯示影像轉換為第二灰階化影像。接著,如步驟S750所示,透過數學型態學(Mathematical Morphology,亦有稱電腦型態學或人體型態學)以及特徵圖像(Feature Patterns)解析出第二灰階化影像中的手部影像。由於人類的手具有固定的型態,故特定的手部動作屬於有限的組合,可內建於資料庫中予以分析比對。然後,如步驟S775所示,對手部影像進行特徵點分析,以擷取手部影像為手勢影像,並將手勢影像製作成手勢點雲。For the gesture analysis, please refer to FIG. 8 and cooperate with FIG. 3 and FIG. 4, wherein FIG. 8 shows a detailed flowchart of step S700 shown in FIG. As shown in FIG. 3, FIG. 4 and FIG. 8, the step S700 of the synchronous positioning and map construction and gesture recognition method of the present case includes the detailed process steps as follows. As shown in step S725, the gesture image capturing unit 25 converts the display image into a second grayscale image. Then, as shown in step S750, the hand in the second grayscale image is analyzed through Mathematical Morphology (also referred to as computer morphology or human body morphology) and feature patterns (Feature Patterns). image. Since human hands have a fixed pattern, specific hand movements are a limited combination that can be built into the database for analysis and comparison. Then, as shown in step S775, the feature image is analyzed by the hand image to capture the hand image as a gesture image, and the gesture image is made into a gesture point cloud.
最後,在細部手勢辨識方面,請參閱第9圖並配合第3圖及第4圖,其中第9圖係顯示第3圖所示之步驟S800之細部流程圖如第3圖、第4圖及第9圖所示,本案同步定位與地圖建構及手勢辨識方法之步驟S800係包括細部流程步驟如下。如步驟S825所示,分析並追蹤前段段末所述之手勢點雲,並根據追蹤結果識別手指、手腕及手掌,以取得手部資訊。接著,如步驟S850所示,根據連續的手部資訊建構立體動作模型,亦即在連續的時間上持續取得手部資訊並根據該等手部資訊的變化建構立體動作模型及其動作。最後,如步驟S875所示,根據立體動作模型進行細部手勢辨識。Finally, in terms of detailed gesture recognition, please refer to FIG. 9 and cooperate with FIG. 3 and FIG. 4, wherein FIG. 9 shows a detailed flowchart of step S800 shown in FIG. 3, as shown in FIG. 3 and FIG. As shown in Fig. 9, the step S800 of the synchronous positioning and map construction and gesture recognition method of the present case includes the detailed process steps as follows. As shown in step S825, the gesture point cloud described in the previous paragraph is analyzed and tracked, and the finger, the wrist and the palm are identified according to the tracking result to obtain the hand information. Next, as shown in step S850, the stereoscopic motion model is constructed based on the continuous hand information, that is, the hand information is continuously acquired for a continuous period of time, and the stereoscopic motion model and its motion are constructed based on the changes of the hand information. Finally, as shown in step S875, detailed gesture recognition is performed according to the stereoscopic motion model.
其中,本案同步定位與地圖建構及手勢辨識方法為提供精確的細部手部辨識,在上述之手部資訊中係可取得包括對每一手指的每一指節之分析資訊,故在判斷手勢上除判斷手掌及手指的閉合、滑動及甩動外,更可判斷不同手指的指節彎曲或伸直狀態,俾以排列組合而能實現多種多樣的手勢操作動作,深富多元彈性。Among them, the synchronous positioning and map construction and gesture recognition method of the present case provide accurate detailed hand recognition, and in the above hand information, the analysis information including each knuckle of each finger can be obtained, so the judgment gesture is In addition to judging the closing, sliding and swaying of the palm and fingers, it is also possible to judge the bending or straightening state of the knuckles of different fingers, and to realize a variety of gesture operation actions by arrangement and combination, and to be rich in multiple elasticity.
另一方面,就未來擴充性而言,由於習知頭戴式電子裝置採用分別擷取目標手勢及目標場景以進行空間定位及手勢辨識之方法,其擷取到的皆為部分影像,甚至可能直接擷取灰階或黑白影像,由於其來源影像非如本案為包含所有資訊的全彩影像,於擴充性上自然不及於本案,且本案更能進一步將包含所有資訊的全彩影像作為來源影像提供給多種多樣的應用程式,於未來擴充性上具有無限可能。On the other hand, in terms of future scalability, since the conventional head-mounted electronic device adopts a method of separately capturing a target gesture and a target scene for spatial positioning and gesture recognition, all of the captured images are partial images, and may even be Directly capture grayscale or black-and-white images. Since the source image is not a full-color image containing all the information, the expansion is naturally not as good as the case, and the case can further use the full-color image containing all the information as the source image. Available in a wide variety of applications, there is unlimited possibilities for future scalability.
綜上所述,本案提供一種同步定位與地圖建構及手勢辨識方法,採用單一的影像擷取模組擷取同時包括目標場景及目標手勢之一擷取影像,並透過動態影像過濾單元及手勢影像擷取單元等影像前處理單元針對各個演算法作前處理並產生顯示影像,最後再由空間定位演算法單元以空間定位演算法進行空間定位,並由細部手勢辨識演算法單元以手勢辨識演算法進行細部手勢辨識,不僅可有效地減少產品體積與重量,同時增進使用者體驗並達成產品微小化的目的,更可進一步降低製造成本。同時,透過上述單一影像擷取模組擷取同時包括目標場景及目標手勢的全彩影像,其影像內容完整且可提供各類軟硬體進行影像處理及後續應用,具有良好的擴充性。In summary, the present invention provides a method for synchronous positioning, map construction and gesture recognition, which uses a single image capture module to capture one of the target scene and the target gesture, and captures the image through the motion image filtering unit and the gesture image. The image pre-processing unit such as the capture unit performs pre-processing on each algorithm and generates a display image. Finally, the spatial positioning algorithm unit performs spatial localization by the spatial positioning algorithm, and the detailed gesture recognition algorithm unit uses the gesture recognition algorithm. The detailed gesture recognition not only can effectively reduce the volume and weight of the product, but also enhance the user experience and achieve the purpose of miniaturization of products, and further reduce the manufacturing cost. At the same time, through the single image capturing module, the full color image including the target scene and the target gesture is captured, and the image content is complete and can provide various software and hardware for image processing and subsequent application, and has good expandability.
縱使本發明已由上述之實施例詳細敘述而可由熟悉本技藝之人士任施匠思而為諸般修飾,然皆不脫如附申請專利範圍所欲保護者。The present invention has been described in detail by the above-described embodiments, and may be modified by those skilled in the art, without departing from the scope of the appended claims.
1‧‧‧習知頭戴式電子裝置1‧‧‧Knowledge head-mounted electronic device
10‧‧‧控制單元 10‧‧‧Control unit
11‧‧‧第一影像擷取模組 11‧‧‧First image capture module
12‧‧‧手勢影像處理器 12‧‧‧ gesture image processor
13‧‧‧第二影像擷取模組 13‧‧‧Second image capture module
14‧‧‧場景影像處理器 14‧‧‧ Scene Image Processor
15‧‧‧手勢辨識演算法集成單元 15‧‧‧ gesture recognition algorithm integration unit
16‧‧‧空間定位演算法集成單元 16‧‧‧ Spatial Positioning Algorithm Integration Unit
2‧‧‧頭戴式電子裝置 2‧‧‧ head mounted electronic devices
20‧‧‧中央處理單元 20‧‧‧Central Processing Unit
21‧‧‧影像擷取模組 21‧‧‧Image capture module
22‧‧‧影像處理單元 22‧‧‧Image Processing Unit
23‧‧‧動態影像過濾單元 23‧‧‧Dynamic image filtering unit
24‧‧‧空間定位演算法單元 24‧‧‧ Spatial Positioning Algorithm Unit
25‧‧‧手勢影像擷取單元 25‧‧‧ gesture image capture unit
26‧‧‧細部手勢辨識演算法單元 26‧‧‧Detailed Gesture Recognition Algorithm Unit
27‧‧‧記憶體單元 27‧‧‧ memory unit
28‧‧‧微投影顯示單元 28‧‧‧Micro-projection display unit
第1圖係顯示習知頭戴式電子裝置之架構方塊圖。 第2圖係顯示習知頭戴式電子裝置擷取目標手勢及目標場景並進行演算之示意圖。 第3圖係顯示本案較佳實施例之同步定位與地圖建構及手勢辨識方法之流程圖。 第4圖係顯示適用本案同步定位與地圖建構及手勢辨識方法之頭戴式電子裝置之架構方塊圖。 第5圖係顯示以第4圖所示之頭戴式電子裝置擷取擷取影像並進行演算之示意圖。 第6圖係顯示第3圖所示之步驟S500之細部流程圖。 第7圖係顯示第3圖所示之步驟S600之細部流程圖。 第8圖係顯示第3圖所示之步驟S700之細部流程圖。 第9圖係顯示第3圖所示之步驟S800之細部流程圖。Figure 1 is a block diagram showing the architecture of a conventional head mounted electronic device. FIG. 2 is a schematic diagram showing a conventional head-mounted electronic device capturing target gestures and target scenes and performing calculations. Figure 3 is a flow chart showing the method of synchronous positioning, map construction and gesture recognition in the preferred embodiment of the present invention. Figure 4 is a block diagram showing the architecture of a head-mounted electronic device that is suitable for the simultaneous positioning, map construction and gesture recognition methods of the present invention. Fig. 5 is a schematic diagram showing the capture of an image and calculation by the head mounted electronic device shown in Fig. 4. Fig. 6 is a flow chart showing the details of step S500 shown in Fig. 3. Fig. 7 is a flow chart showing the details of step S600 shown in Fig. 3. Fig. 8 is a detailed flow chart showing the step S700 shown in Fig. 3. Fig. 9 is a flow chart showing the details of step S800 shown in Fig. 3.
Claims (10)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
TW107107503A TWI648556B (en) | 2018-03-06 | 2018-03-06 | Slam and gesture recognition method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
TW107107503A TWI648556B (en) | 2018-03-06 | 2018-03-06 | Slam and gesture recognition method |
Publications (2)
Publication Number | Publication Date |
---|---|
TWI648556B true TWI648556B (en) | 2019-01-21 |
TW201939105A TW201939105A (en) | 2019-10-01 |
Family
ID=65803870
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
TW107107503A TWI648556B (en) | 2018-03-06 | 2018-03-06 | Slam and gesture recognition method |
Country Status (1)
Country | Link |
---|---|
TW (1) | TWI648556B (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TWI748439B (en) * | 2019-06-27 | 2021-12-01 | 大陸商浙江商湯科技開發有限公司 | Positioning method and device based on shared map, electronic equipment and computer readable storage medium |
TWI838967B (en) * | 2021-12-06 | 2024-04-11 | 宏達國際電子股份有限公司 | Tracking device, head-mounted display device, and virtual reality system |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TWI760978B (en) * | 2019-12-20 | 2022-04-11 | 美商尼安蒂克公司 | Method for locating a camera from images captured by the camera and related non-transitory computer-readable storage medium and computer system |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130147686A1 (en) * | 2011-12-12 | 2013-06-13 | John Clavin | Connecting Head Mounted Displays To External Displays And Other Communication Networks |
TW201706840A (en) * | 2015-06-12 | 2017-02-16 | 英特爾股份有限公司 | Facilitating dynamic runtime transformation of graphics processing commands for improved graphics performance at computing devices |
CN106663411A (en) * | 2014-11-16 | 2017-05-10 | 易欧耐特感知公司 | Systems and methods for augmented reality preparation, processing, and application |
CN106873760A (en) * | 2015-12-14 | 2017-06-20 | 技嘉科技股份有限公司 | Portable virtual reality system |
CN106937531A (en) * | 2014-06-14 | 2017-07-07 | 奇跃公司 | Method and system for producing virtual and augmented reality |
TWI633500B (en) * | 2017-12-27 | 2018-08-21 | 中華電信股份有限公司 | Augmented reality application generation system and method |
-
2018
- 2018-03-06 TW TW107107503A patent/TWI648556B/en active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130147686A1 (en) * | 2011-12-12 | 2013-06-13 | John Clavin | Connecting Head Mounted Displays To External Displays And Other Communication Networks |
CN106937531A (en) * | 2014-06-14 | 2017-07-07 | 奇跃公司 | Method and system for producing virtual and augmented reality |
CN106663411A (en) * | 2014-11-16 | 2017-05-10 | 易欧耐特感知公司 | Systems and methods for augmented reality preparation, processing, and application |
TW201706840A (en) * | 2015-06-12 | 2017-02-16 | 英特爾股份有限公司 | Facilitating dynamic runtime transformation of graphics processing commands for improved graphics performance at computing devices |
CN106873760A (en) * | 2015-12-14 | 2017-06-20 | 技嘉科技股份有限公司 | Portable virtual reality system |
TWI633500B (en) * | 2017-12-27 | 2018-08-21 | 中華電信股份有限公司 | Augmented reality application generation system and method |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TWI748439B (en) * | 2019-06-27 | 2021-12-01 | 大陸商浙江商湯科技開發有限公司 | Positioning method and device based on shared map, electronic equipment and computer readable storage medium |
TWI838967B (en) * | 2021-12-06 | 2024-04-11 | 宏達國際電子股份有限公司 | Tracking device, head-mounted display device, and virtual reality system |
Also Published As
Publication number | Publication date |
---|---|
TW201939105A (en) | 2019-10-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110047104B (en) | Object detection and tracking method, head-mounted display device, and storage medium | |
CN109146965B (en) | Information processing apparatus, computer readable medium, and head-mounted display apparatus | |
CN111738220A (en) | Three-dimensional human body posture estimation method, device, equipment and medium | |
WO2023071964A1 (en) | Data processing method and apparatus, and electronic device and computer-readable storage medium | |
CN114245905A (en) | Depth aware photo editing | |
RU2708027C1 (en) | Method of transmitting motion of a subject from a video to an animated character | |
AU2021290132C1 (en) | Presenting avatars in three-dimensional environments | |
TWI648556B (en) | Slam and gesture recognition method | |
JPWO2005119591A1 (en) | Display control method and apparatus, program, and portable device | |
CN103578135A (en) | Virtual image and real scene combined stage interaction integrating system and realizing method thereof | |
CN111488056B (en) | Manipulating virtual objects using tracked physical objects | |
WO2023173668A1 (en) | Input recognition method in virtual scene, device and storage medium | |
JP7483979B2 (en) | Method and apparatus for playing multi-dimensional responsive images | |
WO2023168957A1 (en) | Pose determination method and apparatus, electronic device, storage medium, and program | |
CN203630822U (en) | Virtual image and real scene combined stage interaction integrating system | |
CN116848556A (en) | Enhancement of three-dimensional models using multi-view refinement | |
CN116580169B (en) | Digital man driving method and device, electronic equipment and storage medium | |
WO2019114092A1 (en) | Image augmented reality method and apparatus, and augmented reality display device and terminal | |
CN116686006A (en) | Three-dimensional scan registration based on deformable model | |
JP6461394B1 (en) | Image generating apparatus and image generating program | |
TWI779332B (en) | Augmented reality system and display method for anchor virtual object thereof | |
US12013985B1 (en) | Single-handed gestures for reviewing virtual content | |
WO2022266556A1 (en) | Methods and systems for motion prediction | |
JP2019139754A (en) | Image generation device and image generation program | |
CN117788758A (en) | Color extraction method, apparatus, electronic device, storage medium and computer program product |