TWI488068B - Gesture control method and apparatus - Google Patents

Description

Gesture control method and device

The present invention relates to a gesture control method and apparatus, and more particularly to a method and apparatus for performing gesture control in a three-dimensional space.

Nowadays, the consumer's requirements for the appearance of electronic devices tend to be light, thin, short, and small, and this requirement makes the size and weight of the electronic device quite limited. Recently, the application of the touch screen has become more and more popular. For a portable device such as a smart phone or a tablet computer, the touch screen can simultaneously serve as a display and input interface of the device, thereby eliminating the cost and configuration of the conventional keyboard. Occupied area. For desktop computers or laptops, consumers can choose to use traditional keyboards or touch screens for control input.

However, in the existing touch screen, whether it is a capacitive touch screen or a resistive touch screen, a touch glass or film is added in front of the liquid crystal display (LCD) panel, thereby increasing the display screen. Thickness and weight. Therefore, when the notebook is turned on, the consumer touches the display screen through the finger, which easily causes the notebook to shake. And the display screen of the notebook computer has a certain distance and angle between the display and the consumer, and is not free to move like a tablet computer, so the device type of the notebook computer is less suitable for direct contact with a finger. In addition, the touch control of the display screen by the finger directly causes the fingerprint to remain on the display screen.

In view of the above, the present invention provides a gesture control method and apparatus, which can correspondingly control an object displayed on a display screen according to a gesture operation performed by a user in a three-dimensional space in front of a display screen.

The invention provides a gesture control method suitable for an electronic device having a display screen. The method first detects the first gesture operation performed by the user in the three-dimensional space before the display screen to define the operation plane. Next, the ratio of the palm of the user relative to the range covered by the operating plane is calculated. Then, according to the ratio, the operation plane is divided into a plurality of operation areas, and the display plane of the display screen is divided into corresponding display areas. Finally, the second gesture operation performed by the user in the operation plane is detected, and the object displayed in the corresponding display area is controlled according to the operation area where the second gesture operation is performed.

In an embodiment of the present invention, the step of controlling an object in the corresponding display area according to the operation area in which the second gesture operation is performed further includes determining a gesture type of the second gesture operation, and performing the gesture type corresponding to the gesture type Control operates on this object.

In an embodiment of the present invention, after the step of detecting a second gesture operation performed by the user in the operation plane, the method further comprises: operating the operation position in the operation plane according to the second gesture, Displays the gesture icon corresponding to the screen position displayed on the screen.

In an embodiment of the present invention, the step of dividing the operation plane according to the proportional division into a plurality of operation areas, and dividing the display plane of the display screen into the corresponding plurality of display areas further comprises one or more according to the ratio The object is displayed in the display area, and the sensing area in each display area for sensing the second gesture operation is defined.

In an embodiment of the present invention, the step of controlling the object displayed in the corresponding display area according to the operation area in which the second gesture operation is performed further comprises determining, according to the ratio, the corresponding gesture of the second gesture operation in the display area. Whether the area includes the sensing area to determine whether to control the object displayed in the display area according to the second gesture operation.

In an embodiment of the invention, the step of detecting the first gesture operation performed by the user in the three-dimensional space before the display screen to define the operation plane includes detecting the viewing distance between the user and the display screen and using The range of extension of the hands is used to determine the position and extent of the operating plane that is suitable for the user to perform the gesture operation.

In an embodiment of the present invention, after the step of detecting a second gesture operation performed by the user in the operation plane, the method further includes detecting a distance between the second gesture operation and the operation plane, And compared with the preset value. If the distance is greater than the preset value, a prompt message is displayed on the display screen to prompt the user to adjust the position of the second gesture operation to be suitable for performing a gesture operation on the operation plane.

In an embodiment of the invention, the step of detecting the second gesture operation performed by the user in the operation plane further comprises detecting whether the second gesture operation moves to a lower edge of the operation plane. If so, the control mode of the electronic device is switched from the gesture control mode to the key input mode.

In an embodiment of the present invention, after the step of switching the control mode of the electronic device from the gesture control mode to the key input mode, the method further comprises: detecting a button operation of the user on the physical keyboard, or projecting the virtual The keyboard detects the key operation of the user on the virtual keyboard to perform key input.

The invention further provides a gesture control device, which comprises a detection module, an operation plane definition module, a gesture operation corresponding module and a control module. The detection module is configured to detect a gesture operation performed by the user before displaying the screen. The operation plane definition module defines an operation plane according to the first gesture operation detected by the detection module. The gesture operation corresponding module is used to calculate the ratio of the palm of the user relative to the range covered by the operation plane, and divides the operation plane into a plurality of operation areas according to the ratio, and the display plane of the split display screen corresponds to Multiple display areas. The control module is configured to determine an operation area of the second gesture detected by the detection module in the operation plane, thereby controlling an object displayed in the corresponding display area.

In an embodiment of the invention, the detecting module further includes detecting a distance between the second gesture operation and the operation plane. The gesture control device further includes a prompting module, configured to compare the distance detected by the detecting module with a preset value, and if the distance is greater than the preset value, the prompting module displays a prompt message on the display screen, The user is prompted to adjust the position of the second gesture operation to be suitable for performing a gesture operation on the operation plane.

In an embodiment of the present invention, the gesture control device further includes a button input module for detecting a button operation of the user on the physical keyboard or detecting the user's projection in the button input mode. Key operations on the virtual keyboard to perform the corresponding key input.

Based on the above, the gesture control method and device provided by the present invention can detect the object displayed in the corresponding display area by detecting the gesture operation performed by the user in the three-dimensional space before the display screen, thereby avoiding direct contact with the finger. Displaying the screen reduces the shaking of the notebook and prevents fingerprints from remaining on the display screen.

The above described features and advantages of the present invention will be more apparent from the following description.

In order to enable the user to perform the manipulation input without directly touching the display screen of the electronic device through the finger, the present invention configures the gesture control device on the electronic device to detect the gesture performed by the user in the three-dimensional space before the display screen. The operation controls and controls the objects displayed on the display screen by detecting the position and type of the gesture operation. In order to clarify the content of the present invention, the following examples are given as examples in which the present invention can be implemented.

FIG. 1 is a block diagram of a gesture control apparatus according to an embodiment of the invention. Referring to FIG. 1 , the gesture control device 100 of the present embodiment is applicable to an electronic device having a display screen, such as a desktop computer or a notebook computer, and is not limited herein. The gesture control device 100 includes a detection module 110, an operation plane definition module 120, a gesture operation corresponding module 130, and a control module 140. The functions are described as follows:

The detection module 110 is, for example, an image sensor having a Charge Coupled Device (CCD) or a Complementary Metal-Oxide Semiconductor (CMOS) component for capturing an image in front of the display screen. To detect the position and type of gestures displayed by the user in front of the screen in three-dimensional space.

The operation plane definition module 120 is coupled to the detection module 110 for analyzing the image captured by the image sensor, thereby obtaining the viewing distance between the user and the display screen and the stretching of the user's hands. The range determines the position and extent of the operating plane that is suitable for the user to perform the gesture operation. In an embodiment, the operation plane definition module 120 may further include a projection unit for projecting a virtual operation plane between the user and the display screen, so that the user can perform a gesture operation on the virtual operation plane.

The gesture operation corresponding module 130 is, for example, an arithmetic unit composed of logical circuit elements for calculating a ratio of a user's palm to an operation plane defined by the operation plane definition module 120, and dividing the operation according to the ratio The plane is a plurality of operation areas, and the display plane of the divided display screen is a corresponding plurality of display areas.

The control module 140 is, for example, a processor or a programmable controller having an arithmetic function, and determines which operation region of the user's hand is located in the operation plane according to the gesture operation position detected by the detection module 110. To control the objects displayed in the corresponding display area.

2 is a flow chart of a gesture control method according to an embodiment of the invention. Referring to FIG. 2, the mode of the embodiment is applied to the gesture control device 100 in the above embodiment. The detailed steps of the gesture control method in this embodiment are described below with the modules in the gesture control device 100.

First, as described in step S210, the detecting module 110 first detects a first gesture operation performed by the user in a three-dimensional space before the display screen to define an operation plane. 3(a) and 3(b) are schematic diagrams showing an application scenario of a gesture control method according to an embodiment of the invention. Referring to FIG. 3( a ) and FIG. 3( b ), the electronic device of the embodiment is, for example, a notebook computer 10 , and the gesture control device 100 is disposed in the notebook computer 10 , wherein the detection module 110 is configured, for example. The image sensor at position A can be used to capture multiple images of the user 20 performing gesture operations.

The first gesture operation referred to in this step is a simple up, down, left, and right swing by the user's hands to define an appropriate operation plane. As shown in Fig. 3(a), the user's 20 hands swing up and down; as shown in Fig. 3(b), the hands of the user 20 are stretched left and right. The operation plane definition module 120 analyzes the image captured by the detection module 110, and obtains the viewing distance between the user 20 and the display screen (the viewing distance d1 of the embodiment is about 600 mm) and the user. 20 hands extended range. In this embodiment, the operation plane defining module 120 defines that the width d2 of the operation plane 30 is 470 mm according to the above condition, the length d3 of the operation plane 30 is 770 mm, and the distance d4 between the operation plane 30 and the display screen. It is 400 mm. The dimension definition of the operation plane is related to the viewing distance and the extension range of the hands. However, the size setting of the operation plane can be adjusted according to actual conditions, and is not limited herein.

Next, in step S220, the gesture operation corresponding module 130 calculates the ratio of the palm of the user relative to the range covered by the operation plane. FIG. 4 is a schematic diagram showing the division of the operation plane and the display plane according to an embodiment of the invention. Referring to FIG. 4, it is assumed that the length d5 of the palm of the user 20 is about 100 mm, and the width d6 of the palm of the user 20 is about 170 mm. Therefore, the length d5 of the palm of the user and the length d3 of the operation plane 30 are The ratio is approximately 1:8; the ratio of the width d6 of the user's palm to the width d2 of the operating plane 30 is approximately 1:3.

After the ratio is obtained, the step S230 is continued. The gesture operation corresponding module 130 divides the operation plane 30 into a plurality of operation areas according to the ratio, and divides the display plane of the display screen into corresponding display areas. As shown in FIG. 4, the operation plane 30 is divided into 24 operation areas 30a to 30x; and the display plane 102 of the notebook computer 10 is also divided into 24 display areas 102a to 102x according to the same ratio.

In addition, after the display areas 102a-102x of the display plane 102 are separated, the gesture operation corresponding module 130 displays one or more objects in the display areas 102a-102x of the display plane 102 according to the above ratio. The object is, for example, an application thumbnail, an image, etc., and is not limited to the above. FIG. 5 is an enlarged schematic view of the display plane 102 of FIG. 4. In the present embodiment, the display plane 102 displays, for example, eight objects I1 to I8. If each of the display areas 102a to 102x displays an object, up to 24 objects can be displayed.

Finally, in step S240, the detecting module 110 detects a second gesture operation performed by the user in the operation plane, and the control module 140 determines an operation area where the second gesture operation is located, thereby controlling the corresponding display area. The object shown. In detail, the gesture operation corresponding module 130 first defines a sensing area for sensing the second gesture operation in each display area according to the size of each display area. The control module 140 determines whether the corresponding gesture area in the display area of the second gesture operation includes a sensing area to determine whether to control the object displayed in the display area according to the second gesture operation.

Taking FIG. 5 as an example, assuming that the display plane 102 shown in FIG. 5 has a size of 11.6 inches, the length d7 of the display plane 102 is about 256 mm; and the width d8 of the display plane 102 is about 144 mm. Therefore, the size of each display area 102a-102x is about 32*48 mm; the size of the sensing area Z is 24*36 mm; and the size of the object is 16*24 mm. It should be noted that a gap must be reserved between the objects in the display plane 102 to avoid malfunction of the device when the gesture area includes two objects at the same time. For example, the distance g1 between the object I2 and the object I3 of the present embodiment is 8 mm; and the distance g2 between the object I3 and the object I4 is 12 mm.

In this embodiment, the ratio of the gesture area GA to the sensing area Z is more than half. Therefore, the control module 140 can control, for example, the object I7 corresponding to the position to be in a reversed form, thereby prompting the user to operate the second gesture. The position is within the sensing range of the object I7. Conversely, if the gesture area GA includes one part of the sensing area Z, but the ratio is less than half, the control module 140 determines that the position of the second gesture operation is not within the sensing range of the object I7, so the object I7 will not Will appear in reverse.

The control module 140 determines whether the position of the second gesture operation is after the sensing range of the object, and further includes determining the type of the gesture of the second gesture operation, and executing the control corresponding to the gesture type to operate the object. Among them, the control operations include zooming in, zooming out, selecting or moving, and the like. FIG. 6 is a schematic diagram of the types of gestures according to an embodiment of the invention. Referring to FIG. 6, gestures 610 to 670 respectively represent different control operations. For example, gesture 640 represents a control operation of "select object"; gesture 670 represents a control operation of "determine execution." The correspondence between the type of the gesture and the control operation can be set by the user in advance, and is not limited herein.

It should be noted that the object of the display plane 102 in the above embodiment is a plurality of different application thumbnails. However, the object displayed on the display plane 102 can also be a single object, such as an image. If the object displayed on the display plane 102 is only one image, the control module 140 can display the image portion in the corresponding display area on the screen according to the type of the gesture operated by the second gesture and the operation area in which the second gesture is operated. The object operates. For example, if the control module 140 determines that the gesture type of the second gesture operation is an enlarged gesture, the control module 140 enlarges the entire image centering on the image portion in the display area.

In this way, the gesture control method and apparatus of the present invention can control the object displayed in the corresponding display area by detecting the gesture operation performed by the user in the three-dimensional space before the display screen, thereby avoiding direct contact with the finger. Touch the display screen to reduce the shaking of the notebook. In addition to the gesture control mode, the user still needs to perform the key input operation by using the keyboard. In this case, the present invention also provides a corresponding adjustment manner, which will be described in detail below.

FIG. 7 is a block diagram of a gesture control apparatus according to another embodiment of the present invention. The gesture control device 700 of the present embodiment is also applicable to an electronic device having a display screen. The electronic device is, for example, a desktop computer or a notebook computer, and is not limited herein.

In this embodiment, the gesture control device 700 includes a detection module 110, an operation plane definition module 120, a gesture operation corresponding module 130, and a control module 140, and a prompt module 750 and a key input module. 760. The prompt module 750 can display a prompt message on the display screen (not shown). The key input module 760 is configured to detect a user's key operation when the control module 140 switches the control mode from the gesture control mode to the key input mode to perform corresponding key input.

FIG. 8 is a flow chart of a gesture control method according to another embodiment of the invention. The mode of operation of the gesture control device 700 will be described below with reference to FIG. Please refer to FIG. 7 and FIG. 8 at the same time.

First, the detecting module 110 first detects a first gesture operation performed by the user in the three-dimensional space before the display screen to define an operation plane (step S810). Next, the gesture operation corresponding module 130 calculates a ratio of the palm of the user with respect to the range covered by the operation plane (step S820). After the ratio is obtained, the gesture operation corresponding module 130 divides the operation plane into a plurality of operation areas according to the ratio, and divides the display plane of the display screen into a plurality of corresponding display areas (step S830). The details of the above steps S810 to S830 are the same as or similar to the steps S210 to S230 in the foregoing embodiment, and are not described herein.

Next, the detection module 110 detects the distance between the second gesture operation and the operation plane. The prompting module 750 is configured to compare the detected distance of the detecting module 110 with a preset value (step S840). If the distance is greater than the preset value, the prompting module 750 displays a prompt message on the display screen for The user is prompted to adjust the position of the second gesture operation to perform the gesture operation on the operation plane (step S850), and returns to step S840 to continue detecting the second gesture operation and the operation plane by the detection module 110. the distance between. The preset value is smaller than the distance between the operation plane and the display screen, and can be adjusted and set according to actual conditions by those skilled in the art.

If the distance between the second gesture operation and the operation plane is not greater than the preset value, the control module 140 further includes determining, according to the image detected by the detection module 110, whether the second gesture operation is moved to the operation plane. The lower edge (step S860). If not, the control module 140 determines the operation position of the second gesture operation in the operation plane, and displays a gesture icon on the corresponding screen position in the display screen (step S870). FIG. 9 is a schematic diagram of a gesture icon according to another embodiment of the invention. Referring to FIG. 9 , in an embodiment, if the second gesture operation of the user is located at the first position GP1 of the operation plane 30 , the control module 140 displays the first display in the display plane 102 according to the corresponding ratio. The corresponding gesture icon is displayed on the location GP1'. Similarly, if the second gesture operation of the user is located at the second position GP2 of the operation plane 30, the control module 140 also displays the corresponding gesture on the second display position GP2' in the display plane 102 according to the corresponding ratio. icon. In another embodiment, the gesture icon displayed in the display plane 102 corresponding to the position of the second gesture operation may also be a fixed mouse cursor icon or other pattern, which is not limited herein.

After the gesture icon is displayed, the control module 140 may further determine the gesture type of the second gesture operation and perform a control operation corresponding to the gesture category.

Going back to step S860, if the control module 140 determines that the second gesture operation has indeed moved to the lower edge of the operation plane, the control module 140 switches the control mode of the electronic device from the gesture control mode to the key input mode (step S880). In one embodiment, the key input module 760 is configured to detect a user's key operation on a physical keyboard of the electronic device in the key input mode. In another embodiment, the detection module 110 further includes a virtual keyboard, and the key input module 760 can detect a user's key operation on the projected virtual keyboard to perform corresponding key input.

In summary, the present invention controls the displayed object in the corresponding display area by detecting the gesture operation performed by the user in the three-dimensional space before the display screen, thereby avoiding the finger touching the display screen and reducing the note. The shaking of the computer can also avoid the problem of fingerprints remaining on the display screen. In addition, the present invention also provides a method for detecting a gesture control mode and a key input mode transition, which is convenient for the user to select an appropriate control mode according to actual operation conditions.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the invention, and any one of ordinary skill in the art can make some modifications and refinements without departing from the spirit and scope of the invention. The scope of the invention is defined by the scope of the appended claims.

10. . . Notebook computer

100, 700. . . Gesture control device

110. . . Detection module

120. . . Operation plane definition module

130. . . Gesture operation corresponding module

140. . . Control module

102a~102x. . . Display area

20. . . user

30. . . Operating plane

30a~30x. . . Operating area

610~670. . . gesture

750‧‧‧ prompt module

760‧‧‧Key input module

A, GP1, GP2‧‧‧ position

GP1’, GP2’‧‧‧ display location

GA‧‧‧ gesture area

I1~I8‧‧‧ objects

Z‧‧‧ Sensing area

D1~d8‧‧‧ length

G1, g2‧‧‧ spacing

S210~S240‧‧‧ steps of the gesture control method of an embodiment

S810~S880‧‧‧ steps of the gesture control method of another embodiment

FIG. 1 is a block diagram of a gesture control apparatus according to an embodiment of the invention.

2 is a flow chart of a gesture control method according to an embodiment of the invention.

3(a) and 3(b) are schematic diagrams showing an application scenario of a gesture control method according to an embodiment of the invention.

FIG. 4 is a schematic diagram showing the division of the operation plane and the display plane according to an embodiment of the invention.

FIG. 5 is an enlarged schematic view of the display plane 102 of FIG. 4.

FIG. 6 is a schematic diagram of the types of gestures according to an embodiment of the invention.

FIG. 7 is a block diagram of a gesture control apparatus according to another embodiment of the present invention.

FIG. 8 is a flow chart of a gesture control method according to another embodiment of the invention.

FIG. 9 is a schematic diagram of a gesture icon according to another embodiment of the invention.

S210~S240. . . Each step of the gesture control method of an embodiment

Claims (18)

A gesture control method is applicable to an electronic device having a display screen, the method comprising the steps of: detecting a first gesture operation performed by a user in a three-dimensional space in front of the display screen to define an operation plane Calculating a ratio of a palm of the user relative to a range covered by the operation plane; dividing the operation plane into a plurality of operation areas according to the ratio, and dividing a display plane of the display screen to correspond to a plurality of a display area; and detecting a second gesture operation performed by the user in the operation plane, and controlling an object displayed in the corresponding display area according to the operation area where the second gesture operation is performed, wherein The step of dividing the operation plane into the operation areas, and dividing the display plane of the display screen into the corresponding display areas further comprises: displaying the object in one or more display areas according to the ratio . The gesture control method of claim 1, wherein the step of controlling the object in the corresponding display area according to the operation area in which the second gesture operation is performed further comprises: determining one of the second gesture operations a gesture type; and a control operation corresponding to the type of the gesture is performed on the object. The gesture control method of claim 1, wherein after the step of detecting the second gesture operation performed by the user in the operation plane, the method further comprises: According to the second gesture operation, in an operation position in the operation plane, a gesture icon is displayed on a corresponding screen position in the display screen. The gesture control method according to claim 1, wherein the step of dividing the operation plane into the operation areas according to the ratio and dividing the display plane of the display screen into corresponding display areas is further The method includes: defining a sensing area in each of the display areas for sensing the second gesture operation. The gesture control method of claim 4, wherein the step of controlling the object displayed in the corresponding display area according to the operation area in which the second gesture operation is performed further comprises: determining the second hand according to the ratio The potential operation includes whether the corresponding one of the gesture areas in the display area includes the sensing area to determine whether to control the object displayed in the display area according to the second gesture operation. The gesture control method of claim 1, wherein the step of detecting the first gesture operation performed by the user in the three-dimensional space before the display screen to define the operation plane comprises: detecting the The viewing distance between the user and the display screen and a range of extensions of the user's hands determine the position and extent of the operating plane suitable for the user to perform the gesture operation. The method of claim 2, wherein the step of detecting the second gesture performed by the user in the operation plane further comprises: detecting the second gesture operation and the operation plane a distance between, and Comparing with a preset value; and if the distance is greater than the preset value, displaying a prompt message on the display screen for prompting the user to adjust the position of the second gesture operation to be suitable for the operation plane Gesture operation on. The method of claim 2, wherein the detecting the second gesture operation performed by the user in the operation plane further comprises: detecting whether the second gesture operation moves to the operation a lower edge of the plane; and if so, switching the control mode of one of the electronic devices from a gesture control mode to a key input mode. The gesture control method of claim 8, wherein after the step of switching the control mode of the electronic device from the gesture control mode to the key input mode, the method further comprises: detecting the user in an entity A key operation on the keyboard, or projecting a virtual keyboard and detecting the user's key operation on the virtual keyboard to perform a key input. A gesture control device includes: a detection module that detects a gesture operation performed by a user in a three-dimensional space before a display screen; and an operation plane definition module that is detected according to the detection module a first gesture operation defining an operation plane; a gesture operation corresponding module, calculating a ratio of a palm of the user relative to a range covered by the operation plane, and segmenting the scale according to the ratio The operation plane is a plurality of operation areas, and a display plane that divides the display screen is a corresponding plurality of display areas; and a control module determines that a second gesture operation detected by the detection module is An operation area in the operation plane, according to which an object displayed in the corresponding display area is controlled, wherein the gesture operation corresponding module displays the object in one or more display areas according to the ratio. The gesture control device of claim 10, wherein the control module further comprises: determining a gesture type of the second gesture operation and performing a control operation corresponding to the gesture type on the object. The gesture control device of claim 10, wherein the gesture operation corresponding module further comprises an operation position in the operation plane according to the second gesture, and a corresponding one in the display screen A gesture icon is displayed on the screen position. The gesture control device of claim 10, wherein the gesture operation corresponding module defines a sensing area in each of the display areas for sensing the second gesture operation. The gesture control device of claim 13, wherein: the control module determines, according to the ratio, whether the second gesture operation corresponds to a gesture area in the display area to include the sensing area, to determine Whether the object displayed in the display area is controlled in accordance with the second gesture operation. The gesture control device of claim 10, wherein the detection module detects an ornamental distance between the user and the display screen and an extended range of the user's hands, the operation plane definition The module is configured to calculate a position and a range of the operation plane suitable for the user to perform a gesture operation. The gesture control device of claim 10, wherein the detecting module further comprises detecting a distance between the second gesture operation and the operation plane, and the gesture control device further comprises: a prompt The module compares the distance detected by the detection module with a preset value. If the distance is greater than the preset value, a prompt message is displayed on the display screen to prompt the user to adjust the first The position of the second gesture operation is adapted to perform a gesture operation on the operation plane. The gesture control device of claim 10, wherein the control module further comprises: determining whether the second gesture operation moves to a lower edge of the operation plane, and if so, controlling a mode from a gesture control mode Switch to a button input mode. The gesture control device of claim 17, further comprising: a button input module, in the button input mode, detecting a button operation of the user on a physical keyboard, or detecting the use The key operation on a virtual keyboard of the projection to perform a corresponding one-key input.