CN102999158B - Gesture recognition method for interactive system and interactive system - Google Patents

Abstract

The invention provides a gesture recognition method of an interactive system and the interactive system, the interactive system comprises an image sensor, a reflecting mirror and at least one light source, the image sensor is used for capturing an image window which comprises a shading shadow formed by at least one indicator shading the light source and/or the reflecting mirror, the gesture recognition method comprises the following steps: capturing an image window by using the image sensor; capturing shading shadow information in the image window; when the image window is judged to comprise two shading shadows, a first mode is entered, and the position coordinate of the indicator relative to the interactive system is calculated according to the position of the shading shadows in the image window; and entering a second mode when the image window is judged to comprise more than two shading light shadows, and performing gesture recognition according to the interrelation of the shading light shadows in the continuous image window.

Description

Gesture recognition method for interactive system and interactive system

This application is a divisional application of an invention patent application with the filing date of September 28, 2009, the application number of 200910176072.0, and the title of "gesture recognition method and interactive system using the method".

technical field

The present invention relates to an interactive system, and in particular to a gesture recognition method and an interactive system using the method.

Background technique

Please refer to FIG. 1 , which shows a conventional touch control system 9 . The touch control system 9 includes a touch surface 90 and at least two cameras 91 and 92 , and the field of view of the cameras 91 and 92 includes the entire touch surface 90 . When the user touches the touch surface 90 with a finger, the cameras 91 and 92 can capture an image window including the light and shadow of the tip of the finger. The processing unit can calculate the two-dimensional position coordinates of the finger touching the touch surface 90 according to the light-shading position of the fingertip in the image window, and perform corresponding operations on the relative control display according to the change of the two-dimensional position coordinates. action.

However, the operating principle of the touch control system 9 is to calculate the two-dimensional position coordinates of the finger touching the touch surface 90 according to the position of the light shadow of the fingertip in each image window. When the user uses multiple fingers to touch the touch surface 90 , compared to the camera 92 , since the fingers will occlude each other, the image window captured by the camera 92 may not necessarily show the shading light and shadow of all fingertips.

For example, in FIG. 1 , the user uses fingers 81 and 82 to touch the touch surface 90, at this time camera 91 captures image window W ₉₁ , which includes shadows I ₈₁ and I _{82 of fingers 81 and 82} ; however, due to finger 81 and 82 occlude each other with respect to the camera 92 , so the image window W ₉₂ captured by the camera 92 only includes one occluded light and shadow. When the processing unit calculates the two-dimensional position coordinates of the finger touching the touch surface 90 according to the image windows W ₉₁ and W ₉₂ , the two-dimensional position coordinates may not be calculated correctly, resulting in malfunctions.

To solve this problem, two cameras 93 and 94 can be set at the other two corners of the touch surface 90 to capture two image windows W ₉₃ and W ₉₄ , and the processing unit can use the image windows W ₉₁ and W 94 The W ₉₃ respectively calculates the two-dimensional position coordinates of the fingers 81 and 82 touching the touch surface 90 . However, such a solution increases system cost.

Contents of the invention

The present invention provides a gesture recognition method for an interactive system. The interactive system includes an image sensor, a reflective mirror, and at least one light source. And/or the image window that shields the light and shadow formed by the reflective mirror surface, the gesture recognition method includes the following steps: using the image sensor to capture the image window; capturing the shadow light and shadow information in the image window; when Enter the first mode when it is determined that the image window includes two shading lights and shadows, and calculate the position coordinates of the indicator relative to the interactive system according to the positions of the shading lights and shadows in the image window; when it is determined that the image window includes When there are more than two shading lights and shadows, enter the second mode, and perform gesture recognition according to the relationship between the shading lights and shadows in the continuous image window.

The present invention also provides an interactive system, which includes: a reflective mirror surface; an image sensor, used to continuously capture at least one image window of shadow light and shadow formed by an indicator covering the reflective mirror surface; a processing unit , enter the first mode when it is determined that the image window captured by the image sensor includes two shading lights and shadows, and calculate according to the one-dimensional position of the shading lights and shadows of the pointer in the image window The two-dimensional position coordinates of the pointer; when it is determined that the image window captured by the image sensor includes more than two shading lights and shadows, the second mode is entered, and the gesture is performed according to the relationship between the shading lights and shadows in the continuous image windows identify.

In view of this, the purpose of the present invention is to propose a gesture recognition method and an interactive system using the method to solve the problems existing in the above-mentioned existing touch system, which is based on the continuous image window captured by the image sensor Gesture recognition based on the interrelationship between shading light and shadow can solve the problem that the contact point coordinates cannot be calculated correctly due to mutual occlusion of pointers.

The present invention proposes a gesture recognition method for an interactive system. The interactive system includes an image sensor, a reflective element, and at least one light source. The resulting image window that blocks light and shadow. The gesture recognition method includes the following steps: using the image sensor to capture an image window; capturing shading light and shadow information in the image window; determining whether multiple pointers are included according to the shading light and shadow information; and when determining When multiple pointers are included, gesture recognition is performed based on the interrelationship of occluded light and shadow in consecutive image windows.

In an embodiment of the gesture recognition method of the present invention, the occlusion light and shadow information includes an average number of light shadows, an average distance between light and shadows and/or a maximum distance between light and shadows.

In an embodiment of the gesture recognition method of the present invention, the step of performing gesture recognition according to the mutual relationship of shading light and shadow in the continuous image window further includes the following steps: comparing one of the average number of light shadows and the average light shadow distance with a preset threshold value comparison; when the average light and shadow number or the average light and shadow distance is greater than the preset threshold value, perform up, down, left, right, zoom in or zoom out gesture recognition; when the average light and shadow number or the average light and shadow distance is less than the When the threshold value is preset, the rotating gesture is recognized; and the display screen of the image display is updated according to the recognized gesture.

The present invention further provides an interactive system, which includes a light emitting unit, an image sensor and a processing unit. The image sensor is used to continuously capture at least one image window in which the light and shadow formed by the indicator shielding the light-emitting unit are blocked. The processing unit performs gesture recognition according to the interrelationship of occluded light and shadow in continuous image windows captured by the image sensor.

In an embodiment of the interactive system of the present invention, the light emitting unit is an active light source or a passive light source. When the light emitting unit is a passive light source, the light emitting unit includes a reflective mirror and the interactive system further includes at least one active light source.

The present invention further proposes a gesture recognition method for an interactive system, the interactive system includes a light emitting unit and an image sensor, and the image sensor is used to capture the shading light and shadow formed by covering the light emitting unit with a plurality of pointers image window. The gesture recognition method includes the following steps: using the image sensor to continuously capture image windows; and performing gesture recognition according to the interrelationships of multiple shading lights and shadows in the continuous image windows.

In an embodiment of the gesture recognition method of the present invention, the interrelationship between the shading lights and shadows includes the average light-shadow distance change, the maximum light-shadow distance change, and the displacement direction of the shading lights and shadows.

According to the gesture recognition method and the interactive system using the method of the present invention, in the first mode, the interactive system controls the action of the cursor according to the change of the two-dimensional coordinates of the pointer; in the second mode, the interactive system controls the movement of the cursor according to the multiple Update the display screen of the display according to the relationship between the shading light and shadow of each pointer, such as scrolling the display screen (scroll), zooming in and out of objects (zoomin/out), rotating objects (rotation), switching screens or menus, etc. In the gesture recognition method of the present invention and the interactive system using the method, since there is no need to calculate the contact point coordinates of multiple pointers, gestures can be performed correctly even if the pointers are mutually occluded with respect to the image sensor identify.

Description of drawings

FIG. 1 shows a schematic diagram of an existing touch control system;

Figure 2a shows a perspective view of an interactive system according to an embodiment of the present invention;

Fig. 2b shows a schematic diagram of the operation of the interactive system of the first embodiment of the present invention;

Fig. 3a shows a schematic diagram of cursor manipulation using the interactive system of the first embodiment of the present invention;

FIG. 3b shows a schematic diagram of an image window captured by the image sensor of FIG. 3a;

Fig. 4a shows the flowchart of the gesture recognition method of the interactive system according to the embodiment of the present invention;

Figure 4b shows a flowchart of the second mode in Figure 4a;

5a-5d respectively show schematic diagrams of recognizing right/left/down/up gestures in the gesture recognition method of the interactive system according to the first embodiment of the present invention;

5e-5f respectively show schematic diagrams of recognizing zoom-in/zoom-out gestures in the gesture recognition method of the interactive system according to the first embodiment of the present invention;

5g to 5h respectively show schematic diagrams of recognizing rotation gestures in the gesture recognition method of the interactive system according to the first embodiment of the present invention;

Figure 6a shows a schematic diagram of the operation of the interactive system of the second embodiment of the present invention;

6b-6c show schematic diagrams of image windows captured by the image sensor of FIG. 6a;

7a-7b respectively show schematic diagrams of recognizing right/left gestures in the gesture recognition method of the interactive system according to the second embodiment of the present invention;

7c to 7d respectively show schematic diagrams of recognizing zoom-in/zoom-out gestures in the gesture recognition method of the interactive system according to the second embodiment of the present invention; and

7e to 7f respectively show schematic diagrams of recognizing rotation gestures in the gesture recognition method of the interactive system according to the second embodiment of the present invention.

Explanation of reference signs

10, 10' interactive system 100 panels

100a the first side of the panel 100b the second side of the panel

3rd side of 100c panel 4th side of 100d panel

100d 4th mirrors the surface of the 100s panel

11 Light emitting unit 11a reflective mirror surface

121 first light source 121' second mirror image

122 second light source 122' third mirror image

13, 13' image sensor 14 processing unit

15 image displays 150 display screens

151 cursor 20, 20', 20" image window

RS real image space IS virtual image space

T ₈₁ , the contact point of the T indicator T ₈₁ ′, the contact point of the first mirror image of T′

A ₈₁ The angle between the contact point and the third side A ₈₁ ′ The angle between the first mirror image and the third side

R ₈₁ 1st sensing path R ₈₁ ′ second sensing path

I ₈₁ , I ₈₂ first shading light and shadow I ₈₁ ′ second shading light and shadow

I ₁ , I ₂ first shading light and shadow I ₁ ′, I ₂ ′ second shading light and shadow

I ₈₁ ″, I ₈₂ ″ cover light and shadow G ₁ first light and shadow group

G ₂ Centerline of the second light and shadow group C

Sav average light and shadow distance 8 users

81, 82 finger 9 touch system

91~94 camera 90 touch surface

W ₉₁ ~W ₉₄ image window S ₁ ~S ₅ steps

Detailed ways

In order to make the above and other objects, features and advantages of the present invention more apparent, a detailed description will be given below with reference to the accompanying drawings. In addition, it should be noted that in the description of the present invention, the same members are denoted by the same symbols.

Please refer to FIG. 2a and FIG. 2b at the same time. FIG. 2a shows a perspective view of the interactive system 10 according to the embodiment of the present invention, and FIG. 2b shows a schematic diagram of the operation of the interactive system 10 according to the first embodiment of the present invention. The interactive system 10 includes a panel 100 , a light emitting unit 11 , a first light source 121 , a second light source 122 , an image sensor 13 , a processing unit 14 and an image display 15 .

The panel 100 includes a first side 100a, a second side 100b, a third side 100c, a fourth side 100d and a surface 100s. Examples of the panel 100 include a whiteboard or a touch screen.

The light emitting unit 11 is disposed on the surface 100s of the first side 100a of the panel 100 . The light emitting unit 11 can be an active light source or a passive light source. When the light emitting unit 11 is an active light source, it can emit light by itself and the light emitting unit 11 is preferably a line light source. When the light emitting unit 11 is a passive light source, it is used to reflect the light emitted by other light sources (such as the first light source 121 and the second light source 122); at this time, the light emitting unit 11 includes a reflective mirror surface 11a facing the third side 100c of the panel , wherein the reflective mirror surface 11a can be formed with appropriate materials. The first light source 121 is disposed on the surface 100s of the second side 100b of the panel, and preferably emits light toward the fourth side 100d of the panel. The second light source 122 is disposed on the surface 100s of the third side 100c of the panel, and preferably emits light toward the first side 100a of the panel; wherein the first light source 121 and the second light source 122 are preferably active light sources, such as wires light source, but not limited thereto.

Please refer to FIG. 2b again. When the light emitting unit 11 is a passive light source (such as a reflective element), the first light source 121 can map a second mirror image 121' relative to the reflective mirror surface 11a, and the second light source 122 can reflect the second mirror image 121' relative to the reflective mirror surface 11a. A third mirror image 122' is mapped out, and the fourth side 100d of the panel can be mapped to a fourth mirror image 100d' relative to the reflective mirror surface 11a; wherein the light emitting unit 11, the first light source 121, the second light source 122 and the fourth side of the panel 100d jointly define a real image space RS; the light emitting unit 11, the second mirror image 121', the third mirror image 122' and the fourth mirror image 100d' jointly define a virtual image space IS.

The image sensor 13 is disposed at a corner of the panel 100 , in this embodiment, the image sensor 13 is disposed at a corner where the third side 100c and the fourth side 100d of the panel meet. The field of view VA of the image sensor 13 includes at least the real image space RS and the virtual image space IS, for capturing images including the real image space RS, the virtual image space IS, and a pointer (pointer) located in the real image space RS, such as a finger 81. Image window with shaded shadows. In one embodiment, the image sensor 13 includes a lens (or lens group) for adjusting the field of view VA of the image sensor 13, so that the image sensor 13 can capture the real image space RS and Complete image of virtual image space IS. Examples of the image sensor 13 include, but are not limited to, CCD image sensors and CMOS image sensors.

The processing unit 14 is coupled to the image sensor 13 and is used for processing images captured by the image sensor 13 to identify one or more pointers. When only one pointer is recognized, the two-dimensional position coordinates of the pointer touching the panel surface 100s are relatively calculated according to the position of the shaded light and shadow of the pointer in the image window. When multiple pointers are identified, the processing unit 14 performs gesture recognition according to the relationship between the shading light and shadow of the pointers in the image window, and controls the image display to update its display screen according to the recognized gestures. The detailed calculation method Will be described in detail later.

The image display 15 is coupled to the processing unit 14, and a cursor 151 may be displayed on a display screen 150 of the image display 15, as shown in FIG. 2b. The processing unit 14 relatively controls the movement of the cursor 151 on the display screen 150 according to the calculated change in the two-dimensional position coordinates of the pointer touching the panel surface 100s, or according to the multiple shaded lights and shadows in the image window captured by the image sensor 13 The interrelationships between the two elements update the display screen of the display screen 150, such as display screen scrolling, object zooming, object rotation, screen switching or menus, and the like.

In order to clearly show the interactive system of the present invention, in Fig. 2a and Fig. 2b, the panel 100 is independent from the image display 15, but it is not used to limit the present invention, in other embodiments, the panel 100 can also be combined on the display screen 150 of the image display 15 . In addition, when the panel 100 is a touch screen, the display screen 150 of the image display 15 can also be used as the panel 100, and the light emitting unit 11, the first light source 121, the second light source 122 and the image sensor 13 are It is arranged on the surface of the display screen 150 .

It can be understood that although in FIG. 2a and FIG. 2b, the panel 100 is shown as a rectangle and the light emitting unit 11, the first light source 121 and the second light source 122 are shown as three vertically arranged on the panel 100. side, but it is only an embodiment of the present invention and is not intended to limit the present invention. In other embodiments, the panel 100 can be made into other shapes; the light emitting unit 11 , the first light source 121 , the second light source 122 and the image sensor 13 can also be disposed on the panel 100 in other spatial relationships. The spirit of the present invention is to use the image sensor 13 to capture the image window, perform gesture recognition according to the displacement of the shading light and shadow in the image window and the relationship between the shading light and shadow, and relatively update the image display according to the recognized gesture display screen.

first embodiment

Please refer to FIG. 3a and FIG. 3b. FIG. 3a shows a schematic diagram of cursor manipulation using the interactive system 10 according to the first embodiment of the present invention; FIG. 3b shows the image window 20 captured by the image sensor 13 in FIG. 3a schematic diagram. As shown in the figure, when a pointing object, such as the tip of a finger ₈₁ , touches the panel surface 100s in the real image space RS, represented by a contact point T81 here, the pointing object relative to the light emitting unit 11 (reflective in this embodiment) The mirror surface 11a of the component) maps the first mirror image in the virtual image space IS, which is represented by the contact point T ₈₁ ′ here. The image sensor 13 captures the tip image of the pointer according to the first sensing route R ₈₁ to form a first shadow I ₈₁ in the image window 20; and captures the second sensing route R ₈₁ ' according to the second A mirror image of the tip to form a second shadow I ₈₁ ′ in the image window 20, as shown in FIG. 3b. In this embodiment, the processing unit 14 prestores the relative relationship between the one-dimensional position of the shading light and shadow in the image window 20 and the angle between the sensing line and the third side 100c of the panel. Therefore, when the image sensor 13 captures the tip image of the pointer and its first mirror image to form the image window 20, the processing unit 14 can respectively calculate the first included angle according to the one-dimensional position of the shaded light and shadow of the image window 20 A ₈₁ and the second included angle A ₈₁ ′. Next, according to the trigonometric function relationship, the processing unit 14 can obtain the two-dimensional position coordinates of the touch point T ₈₁ where the pointer touches the panel surface 100s.

For example, in one embodiment, the panel surface 100s forms a rectangular coordinate system, the third side 100c serves as the X-axis of the rectangular coordinate system, the fourth side 100d serves as the Y-axis of the rectangular coordinate system, and the image sensor 13 position as the origin. Therefore, the coordinates of the contact point T ₈₁ in the Cartesian coordinate system can be expressed as (distance from the fourth side 100d, distance from the third side 100c). In addition, the distance D ₁ between the first side 100 a and the third side 100 c of the panel is pre-stored in the processing unit 14 . In this way, the processing unit 14 can obtain the two-dimensional position coordinates of the touch point T ₈₁ where the indicator 81 touches the surface 100s of the panel according to the following steps: (a) The processing unit 14 calculates the distance between the first sensing route R ₈₁ and the panel The first included angle A ₈₁ between the third side 100c and the second included angle A ₈₁ ′ between the second sensing route R ₈₁ ′ and the third side 100c of the panel; (b) according to the equation D ₂ =2D ₁ /(tanA ₈₁ +tanA ₈₁ ′) Calculate the distance D ₂ between the touch point T ₈₁ of the pointer 81 and the fourth side 100d of the panel; (c) Calculate the touch point T ₈₁ according to D ₂ ×tanA ₈₁ the y-coordinate. Therefore, the two-dimensional position coordinates of the contact point T ₈₁ is (D ₂ , D ₂ ×tanA ₈₁ ).

Please refer to FIG. 3a and FIG. 3b again, the operation of the interactive system 10 according to the first embodiment of the present invention includes two modes. When the processing unit 14 determines according to the image window 20 captured by the image sensor 13 that only one pointer touches the panel surface 100s, the interactive system 10 is controlled to work in the first mode. In the first mode, the image sensor 13 continuously captures images at a sampling frequency, and the processing unit 14 calculates the one-dimensional position of the indicator 81 in the image window 20 according to the occluded light and shadow of the indicator 81, and calculates the 100 s when the indicator 81 touches the surface of the panel. The two-dimensional position coordinates of the touch point _T81 , and according to the change of the two-dimensional position coordinates of the touch point _T81 , the movement of the cursor 151 on the image display 15 is relatively controlled. For example, when the indicator 81 moves toward the fourth side 100d of the panel, the contact point T ₈₁ ′ of the first mirror image also moves toward the fourth mirror image 100d ′ at the same time. At this time, the shading light and shadow I _{81 corresponding to the indicator and the shading light and shadow I 81 ′} corresponding to the first mirror image in the image window 20 also move toward the left side of the image window 20 . In this way, the processing unit 14 calculates the two-dimensional position coordinates of the touch point T ₈₁ according to the positions of the shading light and shadow I ₈₁ and I ₈₁ ′ in each image window 20, and according to the change of the two-dimensional position coordinates, relatively The cursor 151 on the image display 15 is controlled to move toward the left of the display screen 150 . It can be understood that the relationship between the moving direction of the pointer and the moving directions of the shadows I ₈₁ and I ₈₁ ′ in the image window 20 and the moving direction of the cursor 151 is not limited to the content disclosed in the above-mentioned embodiments, The moving direction of the shading light and shadow I ₈₁ and I ₈₁ ′ and the cursor 151 may be opposite to the moving direction of the pointer according to different processing methods of the software.

When the processing unit 14 determines according to the image window 20 captured by the image sensor 13 that there are multiple pointers touching the panel surface 100s, the interactive system 10 is controlled to work in the second mode. In the second mode, the processing unit 14 no longer calculates the two-dimensional position coordinates of each touch point T ₈₁ according to each image window 20 one by one, but only determines the gesture (gesture) according to the interrelationship of multiple shading lights and shadows in the image window 20 , and update the display screen of the display screen 150 of the image display 15 according to the determined gesture, such as scrolling the screen, zooming in and out of objects, rotating objects, switching screens, or displaying menus.

Please refer to FIG. 4a, which shows a flowchart of the gesture recognition method of the present invention. The method includes the following steps: using an image sensor to capture an image window (step S ₁ ); capturing shading light and shadow information in the image window (step S ₂ ); judging whether the shading light and shadow information includes multiple indicator (step S ₃ ); if not, enter the first mode (step S ₄ ); if yes, enter the second mode (step S ₅ ).

Please refer to FIG. 4 b , which shows the implementation of the second mode in step S ₅ in FIG. 4 a , the shading shadow information includes the average number of shadows, the average distance between shadows and the maximum distance between shadows. The second mode includes the following steps: determining whether one of the average number of light shadows and the average distance between light and shadows is greater than a preset threshold (step _S51 ); if yes, performing rotation gesture recognition according to the mutual relationship between shading light and shadow in continuous image windows (step S ₅₂ ); if not, perform up/down/left/right/zoom in/zoom out gesture recognition according to the interrelationships of shading light and shadow in the continuous image window (step S ₅₃ ); and update the display screen of the image display according to the recognized gesture ( Step _S54 ). It can be understood that, in Fig. 4b, it can be set to perform rotation gesture recognition when the average light and shadow number is less than the preset threshold value, and to perform translation gesture recognition when the average light and shadow number is greater than the preset threshold value; or set to when the average light and shadow distance The rotation gesture recognition is performed when the threshold is smaller than the preset threshold value, and the translation gesture recognition is performed when the average light-shadow distance is greater than the preset threshold value.

In another embodiment, the second mode may only include one step: performing rotation gesture recognition according to the relationship between occluded light and shadow in consecutive image windows. In another embodiment, the second mode may only include one step: performing up/down/left/right/zoom in/zoom out gesture recognition according to the interrelationships of shading lights and shadows in consecutive image windows. That is to say, the second mode of the interactive system can only perform one of the recognition of the rotation gesture or the recognition of the up/down/left/right/zoom in/zoom out gesture.

Please also refer to FIGS. 3a to 4b. When using the interactive system 10 of the first embodiment of the present invention for gesture recognition, first use the image sensor 13 to capture an image to form an image window 20, which includes at least one corresponding to the indication. The shading light shadow I ₈₁ of the object contact point T ₈₁ and at least one shading light shadow I ₈₁ ′ corresponding to the first mirror image contact point T ₈₁ ′ (step S ₁ ). Next, the processing unit 14 retrieves shading information in the image window 20 , such as the average number of shading shadows, the average distance between shadows and the maximum distance between shadows, etc., for use in subsequent steps (step S ₂ ). Next, the processing unit 14 determines whether there are multiple indicators in the image window 20 according to the captured shading light and shadow information (step S ₃ ), wherein each indicator will generate a maximum of 2 shading lights and shadows on the image window 20, so when When more than two shadows appear on the image window 20, it means that there are multiple pointers.

When it is determined that only one pointer is included, as shown in FIG. 3 a and FIG. 3 b , the processing unit 14 controls the interactive system 10 to enter the first mode (step S ₄ ). In the first mode, the processing unit 14 calculates the point of contact of the pointer _touching the panel surface _100s (eg T ₈₁ ), and relatively control the movement of the cursor 151 on the image display 15 according to the change of the two-dimensional position coordinates.

When the processing unit 14 determines that there are multiple pointers touching the panel surface 100s according to the shading light and shadow information, as shown in FIG. 5a to FIG. 5h , the interactive system 10 is controlled to enter the second mode (step S ₅ ). In the second mode, the processing unit 14 performs gesture recognition according to the mutual relationship between shading lights and shadows in the image window 20, and controls the screen update of the screen displayed on the display screen 150 of the image display 15 according to the recognized gesture, for example Perform screen scrolling, zoom in and out of objects or windows, rotate objects, switch screens or display menus, etc.

Referring to FIG. 5 a to FIG. 5 h , the implementation of the second mode will be described next, wherein the light emitting unit 11 is described by taking a passive light source as an example in this description. In addition, it can be understood that what is shown in Fig. 5a to Fig. 5h is only illustrative and not intended to limit the present invention.

Screen scrolling gesture: Please refer to Figs. 5a-5d, when the processing unit 14 determines that there are multiple touch points (such as T ₁ and T ₂ ) according to the shading light and shadow information in the image window 20 captured by the image sensing unit 13 , enter the second mode. Next, the processing unit 14 determines whether the average number of shading lights and shadows in the image window 20 is greater than a preset threshold value, such as 6, or whether the average light-shadow distance Sav is greater than a preset threshold value (step S ₅₁ ). When the average number of shading lights and shadows is not greater than the preset threshold value or the average light and shadow distance Sav is not greater than the preset threshold value, pan gesture recognition is performed (step S ₅₃ ).

When recognizing the translation gesture, firstly group the occluded light and shadow into groups, for example, according to the center line C of the image window 20 as the basis for grouping, so as to distinguish the first light and shadow group G ₁ from the second light and shadow group G ₂ , wherein the first light and shadow group G ₁ may be a real image light and shadow group or a virtual image light and shadow group, and the second light and shadow group G ₂ may be a virtual image light and shadow group or a real image light and shadow group.

For example, in FIGS. 5 a to 5 d, when the average light and shadow number in the image window 20 is not greater than the preset threshold value or the average light and shadow distance Sav is not greater than the preset threshold value, the processing unit 14 performs up/down/left/right gesture recognition (step _S53 ). For example, in FIG. 5a, the processing unit 14 recognizes that both the first light and shadow group _G1 and the second light and shadow group G2 in the image window ₂₀ are moving to the right, so it is determined that the user is performing a gesture of scrolling the display screen to the right/left , so that the corresponding display screen is updated relative to the display screen 150 of the control image display 15 (step S ₅₄ ).

Similarly, in FIG. 5b, the processing unit 14 recognizes that both the first light and shadow group _G1 and the second light and shadow group G2 in the image window ₂₀ are moving to the left, so it is determined that the user is performing scrolling of the display screen to the left/right The gesture, and thus the corresponding display screen update is performed relative to the display screen 150 of the control image display 15 (step S ₅₄ ).

In FIG. 5c, the processing unit 14 recognizes that the average light-shadow distance between the first light-shadow group _G1 and the _second light-shadow group G2 in the image window 20 is gradually increasing, so it is determined that the user is performing the scrolling down/up of the display screen. Gestures, so corresponding display screen updates are performed relative to the control display screen 150 (step S ₅₄ ).

In FIG. 5d , the processing unit 14 recognizes that the average light-shadow distance between the first light-shadow group _G1 and the _second light-shadow group G2 in the image window 20 is gradually decreasing, so it is determined that the user is performing the gesture of scrolling the display screen up/down , so the corresponding display screen is updated relative to the control display screen 150 (step S ₅₄ ).

In another embodiment, when the processing unit 14 determines that there are multiple contact points according to the shading light and shadow information in the image window 20 captured by the image sensing unit 13, it directly performs translation gesture recognition (step S ₅₃ ) instead of Execute step _S51 .

Object zoom-in and zoom-out gestures: Before performing the steps of zooming in and out of objects, the user first forms a single contact point on the panel surface 100s to enter the first mode, and controls the cursor 151 to move to the object O in the first mode, as shown in the figure 3a shown. Next, the user forms a plurality of contact points on the panel surface 100s, as shown in FIGS. 5e-5f. When the processing unit 14 determines that there are multiple contact points (eg T ₁ , T ₂ ) according to the shading light and shadow information in the image window 20 captured by the image sensing unit 13 , it enters into the second mode.

Next, the processing unit 14 determines whether one of the average light-shadow number and the average light-shadow distance Sav in the image window 20 is greater than a preset threshold value (step _S51 ). When the threshold value is preset, the shading lights and shadows are grouped first, for example, according to the center line C of the image window 20 as the basis for grouping, so as to distinguish the first light and shadow group G ₁ from the second light and shadow group G ₂ .

For example, in FIGS. 5e-5f, if the average number of light and shadows in the image window 20 is not greater than the preset threshold value or the average light and shadow distance Sav is not greater than the preset threshold value, the processing unit 14 performs zoom-in/zoom-out gesture recognition (step S ₅₃ ). For example, in FIG. 5e, the processing unit 14 recognizes that the average distance between the first light-shadow group _G1 and the _second light-shadow group G2 in the image window 20 does not change substantially, but the maximum light-shadow distance increases. Therefore, it is determined that the user is performing the operation of moving the object. The gesture of zooming in/zooming out, thus correspondingly updating the display screen relative to the display screen 150 of the control image display 15 (step S ₅₄ ).

For example, in FIG. 5f, the processing unit 14 recognizes that the average distance between the first light-shadow group _G1 and the _second light-shadow group G2 in the image window 20 has substantially no change and the maximum light-shadow distance has decreased. Therefore, it is determined that the user is performing the operation of moving the object. The gesture of zooming out/zooming in, thus correspondingly updating the display screen with respect to the display screen 150 of the control image display 15 (step S ₅₄ ).

In another embodiment, when the processing unit 14 determines that there are multiple contact points according to the shading light and shadow information in the image window 20 captured by the image sensing unit 13, it directly performs zoom-in/zoom-out gesture recognition (step S ₅₃ ). Step S _{51 is} not executed.

In addition, before performing zoom-in/zoom-out gesture recognition, it is also possible to directly perform the second mode without entering the first mode. When identifying, you can directly enter the second mode.

Object rotation gesture: before performing the step of object rotation, the user first forms a single contact point on the panel surface 100s to enter the first mode, and controls the cursor 151 to move to the object O in the first mode, as shown in FIG. 3a Show. Next, the user forms multiple contact points T on the panel surface 100s, as shown in FIGS. When a contact point T is reached, the second mode is entered.

Next, the processing unit 14 determines whether one of the average light shadow number and the average light shadow distance Sav in the image window 20 is greater than a preset threshold (step S51 ). When the average number of shadows is larger than the preset threshold value or the average distance Sav between shadows is larger than the preset threshold value, the grouping of the shadows is not performed, and the rotation direction is directly determined according to the number of shadows displaced toward the two sides of the image window 20 .

For example, in FIG. 5g, the processing unit 14 recognizes that the number of shadows moving to the right in the image window 20 is greater than the number of shadows moving to the left, so it is determined that the user is performing a gesture of rotating the object to the right/left, and thus relatively controls the image display. 15. Update the corresponding display screen (step S ₅₄ ).

For example, in FIG. 5h, the processing unit 14 recognizes that the number of shadows moving to the left in the image window 20 is greater than the number of shadows moving to the right, so it is determined that the user is performing a gesture of rotating the object to the left/right, and thus relatively controls the image display. 15. Update the corresponding display screen (step S ₅₄ ).

In another embodiment, when the processing unit 14 determines that there are multiple contact points according to the shading light and shadow information in the image window 20 captured by the image sensing unit 13, it directly recognizes the rotation gesture (step S ₅₂ ) instead of Execute step _S51 .

In addition, it is also possible to directly enter the second mode without entering the first mode before performing the rotation gesture recognition. For example, when the panel 100 is a touch panel, since the user can directly point to the object, it can directly perform the rotation gesture recognition. Enter the second mode.

Screen switching gesture: the user directly forms multiple contact points T on the panel surface 100s, as shown in Figures 5g-5h, when the processing unit 14 determines the When multiple contact points T are included, enter the second mode directly.

The processing unit 14 directly determines whether to perform screen switching according to the number of shading lights and shadows displaced toward the two sides of the image window 20 . For example, in FIG. 5g and FIG. 5h, the processing unit 14 recognizes that the number of shadows moving to the right or to the left in the image window 20 is greater than the number of shadows moving to the left or to the right, so it is determined that the user is performing a screen switching gesture, Therefore, the corresponding screen switching function is performed relative to the control image display 15 .

Display menu gesture: the user directly forms multiple contact points T on the panel surface 100s, as shown in Figures 5g-5h, when the processing unit 14 determines the When multiple contact points T are included, enter the second mode directly.

The processing unit 14 directly determines whether to display the menu according to the number of shading lights and shadows displaced toward the two sides of the image window 20 . For example, in FIG. 5g, the processing unit 14 recognizes that the number of occlusion light shadows moving to the right in the image window 20 is greater than the number of occlusion light shadows moving to the left; in FIG. 5h, the processing unit 14 recognizes the number of occlusion light shadows moving to the left in the image window 20 If it is greater than the number of shadows moving to the right, it is determined that the user is performing a gesture to display a menu, and thus the corresponding menu is displayed on the control image display 15 .

second embodiment

Please refer to FIGS. 6a to 6c. FIG. 6a shows a schematic diagram of the operation of the interactive system 10' according to the second embodiment of the present invention. The schematic diagram of the image windows 20' and 20" taken. In this embodiment, the interactive system 10' includes a light emitting unit 11, a first light source 121, a second light source 122, and image sensors 13 and 13'. The light emitting unit 11 is active light source, and preferably emit light towards the third side 100c of the panel. The light emitting unit 11, the first light source 121 and the second light source 122 are respectively arranged on the first side 100a, the second side 100b and the fourth side 100d of the panel. Therefore, the image The image window 20 ′ captured by the sensor 13 includes only the first shadows I ₈₁ and I ₈₂ of the tip of the pointer; the image window 20 ″ captured by the image sensor 13 ′ only includes the tip of the pointer. Shade light and shadow I ₈₁ ″ and I ₈₂ ″.

In this embodiment, the processing unit 14 performs gesture recognition according to the interrelationships of multiple shading lights and shadows in the image windows 20 ′ and 20 ″ captured by the image sensors 13 and 13 ′.

Screen scrolling gesture: Please refer to Fig. 7a~7b, when the processing unit 14 determines that there are multiple contact points ( For example, when T ₁ and T ₂ ), enter the second mode. Next, the processing unit 14 determines whether the average light and shadow number in the image windows 20' and 20" is greater than the preset threshold value or determines whether the number of light shadows in the image windows 20' and 20" Whether the average light-shadow distance is greater than a preset threshold (step S ₅₁ ).

When the average light and shadow number in the image windows 20' and 20" is not greater than the preset threshold or the average light and shadow distance is not greater than the preset threshold, the processing unit 14 performs left/right gesture recognition (step _S53 ). For example, FIG. 7a 7b, the processing unit 14 respectively recognizes that the shading lights and shadows in the image windows 20' and 20" are moving to the right or to the left, so it is determined that the user is performing the gesture of scrolling the display screen down/up, so the relative Control the display screen 150 of the image display 15 to update the corresponding display screen (step S ₅₄ ).

In another embodiment, when the processing unit 14 determines that there are multiple contact points according to the shading light and shadow information in the image windows 20 and 20 ″ captured by the image sensing units 13 and 13 ′, it directly performs translation gesture recognition ( Step S ₅₃ ) without executing Step S ₅₁ .

Object zoom-in and zoom-out gestures: before performing the steps of zooming in and out, the user controls the cursor to move to the object to be zoomed in and out. Next, the user forms multiple contact points (such as T ₁ and T ₂ ) on the panel surface 100s, as shown in FIGS. 7c-7d. When the processing unit 14 determines that there are multiple contact points according to the shading light and shadow information in the image windows 20 ′ and 20 ″ captured by the image sensing units 13 and 13 ′, the second mode is entered.

Next, the processing unit 14 determines whether the average number of shadows in the image windows 20 and 20 ″ is greater than a preset threshold or whether the average distance between light and shadows in the image windows 20 and 20 ″ is greater than a preset threshold (step S ₅₁ ). When the average number of lights and shadows in the image window 20 is not greater than the preset threshold value or the average light and shadow distance is not greater than the preset threshold value, the processing unit 14 performs zoom-in/zoom-out gesture recognition (step S ₅₃ ). For example, in FIGS. 7c and 7d, the processing unit 14 respectively recognizes that the average distance between light and shadow in the image window 20' and 20" increases or decreases, and therefore determines that the user is performing a gesture of zooming in/out the object, so the relative control The image display 15 updates the corresponding display screen (step S ₅₄ ).

In another embodiment, when the processing unit 14 determines that there are multiple contact points according to the shading light and shadow information in the image windows 20 ′ and 20 ″ captured by the image sensing units 13 and 13 ′, it directly zooms in/out Gesture recognition (step S ₅₃ ) without performing step S ₅₁ .

In addition, before performing zoom-in/zoom-out gesture recognition, it is also possible to directly perform the second mode without entering the first mode. When identifying, you can directly enter the second mode.

Object rotation gesture: before performing the step of object rotation, the user controls the cursor to move to the object to be rotated. Next, the user forms a plurality of contact points T on the panel surface 100s, as shown in FIGS. When it is determined that the shading light and shadow information includes multiple contact points T, the second mode is entered.

Next, the processing unit 14 determines whether the average light and shadow number in the image windows 20' and 20" is greater than a preset threshold value or whether the processing unit 14 determines whether the average light and shadow distance in the image windows 20' and 20" is greater than a preset threshold value (step S ₅₁ ). When the average number of light shadows is greater than the preset threshold value or the average light shadow distance is greater than the preset threshold value, the rotation direction is determined according to the number of shading light shadows moving toward the two sides of the image windows 20 ′ and 20 ″.

For example, in FIG. 7e, the processing unit 14 recognizes that the number of occlusion light shadows moving to the right in the image windows 20' and 20" is greater than the number of occlusion light shadows moving to the left; in FIG. In 20″, the number of shading lights and shadows moving to the left is greater than the number of shading lights and shadows moving to the right, so it is determined that the user is performing a gesture of rotating the object to the right/left, so the corresponding display screen update is performed on the control image display 15 (step _S54 ).

In another embodiment, when the processing unit 14 determines that there are multiple contact points according to the shading light and shadow information in the image windows 20 ′ and 20 ″ captured by the image sensing units 13 and 13 ′, the rotation gesture recognition is performed directly. (Step S ₅₂ ) Step S ₅₁ is not executed.

In addition, it is also possible to directly perform the second mode without entering the first mode before performing the rotation gesture recognition.

Screen switch gesture or menu display gesture: the user directly forms multiple contact points T on the panel surface 100s, as shown in Fig. When it is determined that the shading light and shadow information in 20' and 20" includes multiple contact points T, it will directly enter the second mode.

The processing unit 14 directly determines whether to perform screen switching or display menu gestures based on the number of shading lights and shadows displaced toward the two sides of the image windows 20' and 20". For example, in Fig. 7e and Fig. 7f, the processing unit 14 recognizes the 20′ and 20″, the number of shading light shadows moving to the right or left is greater than the number of shading light shadows moving to the left or right, so it is determined that the user is performing a gesture of switching screens or displaying menu gestures, and thus proceeding with respect to the control image display 15. Corresponding screen switching or display menu.

It can be understood that the control functions corresponding to the interrelationships of shading light and shadow in the above second mode are not limited to the content disclosed in FIGS. 5a-5h and FIGS. 7a-7f. The spirit of the present invention is to perform gesture recognition based on the interrelationship of shading light and shadow in the image window without calculating the position coordinates of the contact points one by one to avoid the situation that the coordinates of the contact points cannot be calculated due to mutual occlusion of pointers.

As mentioned above, since the existing touch control system performs gesture recognition through the change of the two-dimensional coordinates of the contact points, it is easy to fail to calculate the coordinates of the contact points correctly when the pointers cover each other. The present invention recognizes the interrelationship of shaded light and shadow in the image window as the basis for gesture recognition, so only one image sensor can be used for correct gesture recognition, which has the effect of reducing system cost.

Although the present invention has been disclosed by the above-mentioned embodiments, the above-mentioned embodiments are not intended to limit the present invention, and any skilled person in the technical field to which the present invention belongs should be able to make various changes without departing from the spirit and scope of the present invention with modification. Therefore, the protection scope of the present invention should be determined by the scope defined in the appended claims.

Claims (14)

1. the gesture identification of an interaction systems, described interaction systems comprises an image sensor, a mirror surface and at least one light source, described image sensor comprises at least one corresponding to indicant contact point and at least one viewing windows covering shadow corresponding to mirror image contact point for capturing, and described gesture identification comprises the following steps:

Utilize described image sensor acquire images form;

Capture the shield light shadow information in described viewing windows;

When judge described viewing windows comprise two cover shadow time enter first mode, according to the position calculation of covering shadow in described viewing windows, indicant is relative to the position coordinates of described interaction systems; And

Enter the second pattern when the described viewing windows of judgement comprises when two or more covers shadow, the mutual relationship according to covering shadow in consecutive image form carries out gesture identification,

Wherein, described second pattern comprises:

By average shadow number and average shadow spacing, one of them compares with predetermined threshold level; And

When described average shadow number or described average shadow spacing are less than described predetermined threshold level, the gesture identification carry out upper and lower, left and right, zooming in or out.

2. gesture identification according to claim 1, wherein, described first mode also comprises the following steps:

Position coordinates according to indicant described in consecutive image form changes, the action of cursor on relation control image display.

3. gesture identification according to claim 1, wherein, described second pattern also comprises the following steps:

When described average shadow number or described average shadow spacing are greater than described predetermined threshold level, carry out rotate gesture identification; And

The display frame of image display is upgraded according to picked out gesture;

Wherein, when described gesture is recognized as upper and lower, left and right, described display frame is updated to display frame scrolling; When described gesture be recognized as amplification, reduce time, described display frame is updated to object Scalable; When described gesture is recognized as rotate gesture, described display frame is updated to object and rotates; When described gesture be recognized as move left and right cover shadow number different time, described display frame is updated to picture and switches or display menu.

4. gesture identification according to claim 1, wherein, when described average shadow number or described average shadow spacing are less than predetermined threshold level, described gesture identification also comprises the following steps:

Shadow will be covered according to the center line of described viewing windows and be divided into a real image shadow group and a virtual image shadow group;

Wherein, when the described real image shadow group in described viewing windows and described virtual image shadow group displacement in the same way time, carry out the identification of left or right gesture; When the described real image shadow group in described viewing windows and the average shadow spacing between described virtual image shadow group change, carry out upper or lower gesture identification; When the described real image shadow group in described viewing windows and the maximum shadow spacing between described virtual image shadow group change, zoom in or out gesture identification.

5. gesture identification according to claim 3, wherein, in the step of the gesture identification carrying out rotating, according to displacement in consecutive image form in the same way cover the higher direction recognition sense of rotation of covering shadow of shadow number.

6. an interaction systems, this interaction systems comprises:

At least one active light source;

A mirror surface;

An image sensor, comprises at least one corresponding to indicant contact point and at least one viewing windows covering shadow corresponding to mirror image contact point for capturing continuously; And

Processing unit, when judge the described viewing windows that captures of described image sensor comprise two cover shadow time enter first mode, according to described indicant, cover the one-dimensional position that shadow is arranged in described viewing windows, calculate the two-dimensional position coordinate of described indicant; Enter the second pattern when judging that the described viewing windows that captures of described image sensor comprises when two or more covers shadow, the mutual relationship according to covering shadow in consecutive image form carries out gesture identification,

Wherein, the viewing windows that described image sensor captures comprises real image and covers shadow and shadow is covered in the virtual image, and described processing unit covers according to real image in consecutive image form the mutual relationship that shadow and the virtual image cover shadow and carries out gesture identification.

7. interaction systems according to claim 6, wherein, described interaction systems also comprises the image display coupling described processing unit, and described processing unit also upgrades the display frame of described image display according to picked out gesture.

8. interaction systems according to claim 6, wherein, the mutual relationship of covering shadow in the consecutive image form that described processing unit only captures according to described image sensor is carried out gesture identification and does not calculate the two-dimensional position coordinate of each described indicant one by one according to described viewing windows.

9. the gesture identification of an interaction systems, described interaction systems comprises an image sensor, a mirror surface and at least one light source, described image sensor comprises at least one corresponding to indicant contact point and at least one viewing windows covering shadow corresponding to mirror image contact point for capturing, and described gesture identification comprises the following steps:

Utilize described image sensor acquire images form;

Capture the shield light shadow information in described viewing windows;

When judge described viewing windows comprise two cover shadow time enter first mode, according to the position calculation of covering shadow in described viewing windows, indicant is relative to the position coordinates of described interaction systems; And

Enter the second pattern when the described viewing windows of judgement comprises when two or more covers shadow, the mutual relationship according to covering shadow in consecutive image form carries out gesture identification,

Wherein, described second pattern comprises:

By average shadow number and average shadow spacing, one of them compares with predetermined threshold level; And

When described average shadow number or described average shadow spacing are greater than described predetermined threshold level, carry out rotate gesture identification.

10. gesture identification according to claim 9, wherein, described first mode also comprises the following steps:

Position coordinates according to indicant described in consecutive image form changes, the action of cursor on relation control image display.

11. gesture identifications according to claim 9, wherein, described second pattern also comprises:

The display frame of image display is upgraded according to picked out gesture;

Wherein, when described gesture is recognized as rotate gesture, described display frame is updated to object and rotates.

12. gesture identifications according to claim 9, wherein, in the step of the gesture identification carrying out rotating, according to displacement in consecutive image form in the same way cover the higher direction recognition sense of rotation of covering shadow of shadow number.

The gesture identification of 13. 1 kinds of interaction systems, described interaction systems comprises an image sensor, a mirror surface and at least one light source, described image sensor comprises at least one corresponding to indicant contact point and at least one viewing windows covering shadow corresponding to mirror image contact point for capturing, and described gesture identification comprises the following steps:

Utilize described image sensor acquire images form;

Capture the shield light shadow information in described viewing windows;

When judge described viewing windows comprise two cover shadow time enter first mode, according to the position calculation of covering shadow in described viewing windows, indicant is relative to the position coordinates of described interaction systems; And

Enter the second pattern when the described viewing windows of judgement comprises when two or more covers shadow, the mutual relationship according to covering shadow in consecutive image form carries out gesture identification,

Wherein, described second pattern comprises:

By average shadow number and average shadow spacing, one of them compares with predetermined threshold level; And

When described average shadow number or described average shadow spacing are less than predetermined threshold level, shadow will be covered and be divided into a real image shadow group and a virtual image shadow group, and cover according to real image in consecutive image form the mutual relationship that shadow and the virtual image cover shadow and carry out gesture identification.

14. gesture identifications according to claim 13, wherein, when the described real image shadow group in described viewing windows and described virtual image shadow group displacement in the same way time, carry out the identification of left or right gesture; When the described real image shadow group in described viewing windows and the average shadow spacing between described virtual image shadow group change, carry out upper or lower gesture identification; When the described real image shadow group in described viewing windows and the maximum shadow spacing between described virtual image shadow group change, zoom in or out gesture identification.