CN113158845A - Gesture recognition method, head-mounted display device and nonvolatile storage medium - Google Patents
Gesture recognition method, head-mounted display device and nonvolatile storage medium Download PDFInfo
- Publication number
- CN113158845A CN113158845A CN202110364942.8A CN202110364942A CN113158845A CN 113158845 A CN113158845 A CN 113158845A CN 202110364942 A CN202110364942 A CN 202110364942A CN 113158845 A CN113158845 A CN 113158845A
- Authority
- CN
- China
- Prior art keywords
- hand
- gesture recognition
- image
- head
- mounted display
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 210000003462 vein Anatomy 0.000 claims description 5
- 230000008569 process Effects 0.000 description 10
- 230000009471 action Effects 0.000 description 8
- 230000033001 locomotion Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000000605 extraction Methods 0.000 description 4
- 230000001133 acceleration Effects 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 2
- 239000011521 glass Substances 0.000 description 2
- 230000005484 gravity Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000010079 rubber tapping Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
- G06V40/113—Recognition of static hand signs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/14—Vascular patterns
Abstract
The invention discloses a gesture recognition method, a head-mounted display device and a nonvolatile storage medium, wherein the method comprises the following steps: if a gesture recognition instruction is detected, acquiring a first image in a shooting area in real time, and extracting a first hand feature point in the first image, wherein the first image is an image containing hand features of a user to be recognized; determining a second hand characteristic point corresponding to the first hand characteristic point in a characteristic library of the head-mounted display device, and determining the similarity between the first hand characteristic point and the second hand characteristic point; determining target hands belonging to the wearer in the first image according to the similarity, and determining the number of the target hands corresponding to the target hands; and if the number of the target hands is greater than or equal to a first preset value, executing gesture recognition operation corresponding to the gesture recognition instruction. The invention avoids the interference of external factors including the gestures of non-wearers on the gesture recognition, and improves the accuracy of the gesture recognition.
Description
Technical Field
The invention relates to the technical field of intelligent glasses, in particular to a gesture recognition method, a gesture recognition device, gesture recognition equipment and a nonvolatile storage medium.
Background
The gesture recognition technology is subjected to two-dimensional gesture recognition, and currently enters the development stage of three-dimensional gesture recognition, as the name suggests, the two-dimensional gesture recognition is only carried out in plane dimensions, such as left and right and up and down movement, and the three-dimensional gesture recognition adds a depth dimension.
In the prior art, the existing head-mounted display device recognizes the gesture action in a specific area through a gesture recognition sensor in the head-mounted display device in the gesture action recognition process, and then realizes touchless control. However, the existing head-mounted display device is easily affected by the actions of other non-device wearers in the environment in the process of recognizing the gesture actions, so that the gesture is recognized by mistake, and the problem of false triggering exists in the process of recognizing the gesture actions.
Therefore, how to prevent the interference of external factors to gesture recognition and improve the accuracy of gesture recognition, especially the detection accuracy in the gesture recognition process, is a technical problem that needs to be solved by those skilled in the art at present.
The above is only for the purpose of assisting understanding of the technical aspects of the present invention, and does not represent an admission that the above is prior art.
Disclosure of Invention
The invention mainly aims to provide a gesture recognition method, a gesture recognition device, equipment and a nonvolatile storage medium, and aims to solve the technical problem that gesture misidentification exists because existing head-mounted display equipment is easily influenced by actions of other non-device wearers in the environment in the process of recognizing gesture actions.
In order to achieve the above object, the present invention provides a gesture recognition method applied to a head-mounted display device, the gesture recognition method including the steps of:
if the head-mounted display equipment detects a gesture recognition instruction, acquiring a first image in a shooting area of the head-mounted display equipment in real time, and extracting a first hand feature point in the first image, wherein the first image is an image containing hand features of a user to be recognized;
determining a second hand feature point corresponding to the first hand feature point in a feature library of the head-mounted display device, and determining the similarity between the first hand feature point and the second hand feature point;
determining target hands belonging to a wearer in the first image according to the similarity, and determining the number of the target hands corresponding to the target hands;
and if the number of the target hands is larger than or equal to a first preset value, executing gesture recognition operation corresponding to the gesture recognition instruction.
Optionally, the step of determining a second hand feature point corresponding to the first hand feature point in the feature library of the head-mounted display device includes:
determining the number of hands contained in the first image based on the first hand feature point;
and if the number of the hands is smaller than or equal to a second preset numerical value and larger than a third preset numerical value, determining a second hand characteristic point corresponding to the first hand characteristic point in a characteristic library of the head-mounted display device.
Optionally, after the step of determining the number of hands included in the first image based on the first hand feature point, the method further includes:
if the number of the hands is larger than the second preset value, outputting prompt information that the current gesture contains the hands of the non-wearer and whether the gesture is recognized or not, and receiving feedback information corresponding to the prompt information;
and executing gesture recognition operation corresponding to the gesture recognition instruction based on the feedback information.
Optionally, the step of executing, based on the feedback information, a gesture recognition operation corresponding to the gesture recognition instruction includes:
if the feedback information contains a first execution instruction corresponding to the gesture recognition operation, executing the gesture recognition operation;
and if the feedback information contains a second execution instruction corresponding to the execution of the gesture recognition operation, executing the step of acquiring a first image in a shooting area of the head-mounted display equipment in real time and extracting a first hand feature point in the first image.
Optionally, the step of determining a target hand belonging to the wearer in the first image according to the similarity comprises:
determining the hand with the similarity larger than the preset similarity, and taking the hand with the similarity larger than the preset similarity as a target hand, wherein the target hand is the hand belonging to the wearer in the first image.
Optionally, before the step of determining a second hand feature point corresponding to the first hand feature point in the feature library of the head-mounted display device and determining a similarity between the first hand feature point and the second hand feature point, the method further includes:
acquiring a second image in a shooting area of the head-mounted display device, and extracting a second hand feature point in the second image, wherein the second image is an image containing the hand feature of the wearer;
storing the second hand feature point in a feature library of the head-mounted display device.
Optionally, the second image comprises one or more of a hand frontal unfolded image, a left unfolded image, a right unfolded image, a back unfolded image, a fist frontal image, a fist side image, or a fist back image, and the second hand feature points comprise one or more of a distance between a fingertip point and a valley point, a hand contour size, a hand color depth, a joint point, or a hand vein feature.
Optionally, after the step of executing the gesture recognition operation corresponding to the gesture recognition instruction, the method further includes:
acquiring a gesture recognition result based on the gesture recognition operation, and determining an operation instruction corresponding to the gesture recognition result;
executing the operation instruction on the head-mounted display device.
Further, to achieve the above object, the present invention also provides a head mounted display device including: the gesture recognition system comprises a memory, a processor and a gesture recognition program stored on the memory and capable of running on the processor, wherein the gesture recognition program realizes the steps of the gesture recognition method when being executed by the processor.
In addition, to achieve the above object, the present invention further provides a non-volatile storage medium, in which a gesture recognition program is stored, and the gesture recognition program implements the steps of the gesture recognition method when executed by a processor.
According to the method, if the head-mounted display equipment detects a gesture recognition instruction, a first image in a shooting area of the head-mounted display equipment is collected in real time, and a first hand feature point in the first image is extracted, wherein the first image is an image containing hand features of a user to be recognized; determining a second hand feature point corresponding to the first hand feature point in a feature library of the head-mounted display device, and determining the similarity between the first hand feature point and the second hand feature point; determining target hands belonging to a wearer in the first image according to the similarity, and determining the number of the target hands corresponding to the target hands; and if the number of the target hands is larger than or equal to a first preset value, executing gesture recognition operation corresponding to the gesture recognition instruction. In this embodiment, before the gesture is recognized, a first image is collected, and a first hand feature point in the first image is extracted; comparing the first hand characteristic points with pre-stored second hand characteristic points of the wearer, and calculating the similarity between the first hand characteristic points and the second hand characteristic points to identify whether the currently identified hand is the hand of the wearer; if the similarity between the first hand feature point and the second hand feature point meets a preset condition, which indicates whether the currently recognized hand is the hand of the wearer, and the hand features of the non-wearer in the first image are less, the gesture recognition operation may be performed. This embodiment gathers the current first hand feature of the person of wearing earlier before carrying out the gesture recognition operation and verifies, carries out the gesture recognition operation again to can avoid external factors to include the interference of non-person of wearing's gesture to gesture recognition, promote gesture recognition's accuracy.
Drawings
FIG. 1 is a schematic diagram of a head-mounted display device in a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a gesture recognition method according to a first embodiment of the present invention;
FIG. 3 is a flowchart illustrating a gesture recognition method according to a second embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a head-mounted display device in the gesture recognition method according to the present invention;
fig. 5 is a schematic external structural diagram of a head-mounted display device in the gesture recognition method of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As shown in fig. 1, fig. 1 is a schematic structural diagram of a head-mounted display device in a hardware operating environment according to an embodiment of the present invention.
The head-mounted display device in the embodiment of the present invention may be an Augmented Reality (Augmented Reality) device, or may also be a Virtual Reality (Virtual Reality) or Mixed display (Mixed Reality) device, and the specific implementation mode is developed and explained by taking the head-mounted display device as an example.
As shown in fig. 1, the head-mounted display device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a storage device separate from the processor 1001.
Optionally, the head-mounted display device may further include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, and so on. Such as light sensors, motion sensors, and other sensors. In particular, the light sensor may include an ambient light sensor that may adjust the brightness of the display screen according to the brightness of ambient light, and a proximity sensor that may turn off the display screen and/or backlight when the head-mounted display device is moved to the ear. As one of the motion sensors, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally, three axes), detect the magnitude and direction of gravity when the device is stationary, and can be used for applications of recognizing the posture of the head-mounted display device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer and tapping), and the like; of course, the head-mounted display device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which are not described herein again.
Those skilled in the art will appreciate that the configuration of the head mounted display device shown in FIG. 1 does not constitute a limitation of the head mounted display device, and may include more or less components than those shown, or some components in combination, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is one type of computer storage medium, may include therein an operating system, a network communication module, a user interface module, and a gesture recognition program.
In the head-mounted display device shown in fig. 1, the network interface 1004 is mainly used for connecting to a background server and performing data communication with the background server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and the processor 1001 may be used to invoke a gesture recognition program stored in the memory 1005.
In this embodiment, the head-mounted display apparatus includes: a memory 1005, a processor 1001 and a gesture recognition program stored in the memory 1005 and executable on the processor 1001, wherein the processor 1001, when calling the gesture recognition program stored in the memory 1005, performs the following operations:
if the head-mounted display equipment detects a gesture recognition instruction, acquiring a first image in a shooting area of the head-mounted display equipment in real time, and extracting a first hand feature point in the first image, wherein the first image is an image containing hand features of a user to be recognized;
determining a second hand feature point corresponding to the first hand feature point in a feature library of the head-mounted display device, and determining the similarity between the first hand feature point and the second hand feature point;
determining target hands belonging to a wearer in the first image according to the similarity, and determining the number of the target hands corresponding to the target hands;
and if the number of the target hands is larger than or equal to a first preset value, executing gesture recognition operation corresponding to the gesture recognition instruction.
Further, the processor 1001 may call a gesture recognition program stored in the memory 1005, and also perform the following operations:
determining the number of hands contained in the first image based on the first hand feature point;
and if the number of the hands is smaller than or equal to a second preset numerical value and larger than a third preset numerical value, determining a second hand characteristic point corresponding to the first hand characteristic point in a characteristic library of the head-mounted display device.
Further, the processor 1001 may call a gesture recognition program stored in the memory 1005, and also perform the following operations:
if the number of the hands is larger than the second preset value, outputting prompt information that the current gesture contains the hands of the non-wearer and whether the gesture is recognized or not, and receiving feedback information corresponding to the prompt information;
and executing gesture recognition operation corresponding to the gesture recognition instruction based on the feedback information.
Further, the processor 1001 may call a gesture recognition program stored in the memory 1005, and also perform the following operations:
if the feedback information contains a first execution instruction corresponding to the gesture recognition operation, executing the gesture recognition operation;
and if the feedback information contains a second execution instruction corresponding to the execution of the gesture recognition operation, executing the step of acquiring a first image in a shooting area of the head-mounted display equipment in real time and extracting a first hand feature point in the first image.
Further, the processor 1001 may call a gesture recognition program stored in the memory 1005, and also perform the following operations:
determining the hand with the similarity larger than the preset similarity, and taking the hand with the similarity larger than the preset similarity as a target hand, wherein the target hand is the hand belonging to the wearer in the first image.
Further, the processor 1001 may call a gesture recognition program stored in the memory 1005, and also perform the following operations:
acquiring a second image in a shooting area of the head-mounted display device, and extracting a second hand feature point in the second image, wherein the second image is an image containing the hand feature of the wearer;
storing the second hand feature point in a feature library of the head-mounted display device.
Further, the processor 1001 may call a gesture recognition program stored in the memory 1005, and also perform the following operations:
acquiring a gesture recognition result based on the gesture recognition operation, and determining an operation instruction corresponding to the gesture recognition result;
executing the operation instruction on the head-mounted display device.
The invention also provides a gesture recognition method, and referring to fig. 2, fig. 2 is a schematic flow chart of a first embodiment of the gesture recognition method of the invention.
In this embodiment, the gesture recognition method includes the following steps:
step S10, if the head-mounted display device detects a gesture recognition instruction, acquiring a first image in a shooting area of the head-mounted display device in real time, and extracting a first hand feature point in the first image, wherein the first image is an image containing hand features of a user to be recognized;
the gesture recognition method provided by the invention is applied to wearable head-mounted display equipment, and refers to the structure diagram of the head-mounted display equipment in fig. 4, and the head-mounted display equipment comprises a hand feature point extraction module, a gesture recognition module and a processor. Wherein, hand characteristic point draws the characteristic point that the module is used for discerning and draws the hand, and gesture recognition module is used for discerning the gesture action, and hand characteristic point draws module and treater electric connection, gesture recognition module and treater electric connection, and wearable head-mounted display device can be AR glasses.
Referring to fig. 5, a head-mounted display device as shown in fig. 5, 1 denotes a lens portion of the head-mounted display device, 2 denotes a frame portion of the head-mounted display device, 3 denotes a gesture feature point extraction module in the head-mounted display device, 4 denotes a gesture recognition module, and 5 denotes a side body portion of the head-mounted display device. Further, the gesture recognition module 4 may be disposed outside the housing of the middle connection portion of the two lenses, or disposed inside the housing of the head-mounted display device.
In this embodiment, wear display device can also include the high definition digtal camera module, and the high definition digtal camera module can be two and take a photograph, also can be more. When the gesture is recognized, a first image of a target area is collected through the high-definition camera module, so that an image of a wearer wearing the head-mounted display equipment is collected, wherein the target area is an area where a shooting range of the high-definition camera module is located. When receiving a gesture recognition instruction, the head-mounted display equipment collects images in a shooting area of the head-mounted display equipment to obtain a first image; after the first image of the target area is collected, the high-definition camera module transmits the collected first hand image to a hand feature point extraction module of the head-mounted display device, so that the hand feature point extraction module extracts hand feature points in the first image from the first image, and first hand feature points corresponding to the first image are obtained.
Step S20, determining a second hand feature point corresponding to the first hand feature point in a feature library of the head-mounted display device, and determining a similarity between the first hand feature point and the second hand feature point;
in this embodiment, after the first hand feature point in the first image is extracted, the second hand feature point pre-stored in the feature library is acquired, and the extracted first hand feature point and the second hand feature point are compared in a one-to-one correspondence manner, so as to obtain the similarity between the first hand feature point and the second hand feature point. And the second hand characteristic points pre-stored in the characteristic library are various kinds of hand characteristic information pre-stored by the head-mounted display equipment wearer. In this embodiment, by comparing the first hand feature point and the second hand feature point, the hand feature point (first hand feature point) in the currently recognized first image can be compared with the hand feature point (second hand feature point) of the pre-stored wearer (user), so as to obtain the similarity between the currently recognized hand feature and the hand feature pre-stored by the user.
Specifically, the first hand feature points include a plurality of first hand feature points, the second hand feature points include a plurality of second hand feature points, and the similarity between each first hand feature point and each second hand feature point corresponding to the first hand feature point is obtained by comparing the first hand feature point with the second hand feature point corresponding to the first hand feature point.
Further, the hand feature points include a distance between a fingertip point and a valley point, a hand contour size, a hand color depth, a joint point, or a hand vein feature.
Step S30, determining target hands belonging to the wearer in the first image according to the similarity, and determining the number of the target hands corresponding to the target hands;
step S40, if the number of the target hands is greater than or equal to a first preset value, perform a gesture recognition operation corresponding to the gesture recognition instruction.
In this embodiment, after the similarity between the first hand feature point and the second hand feature point is obtained, the target hand belonging to the wearer in the first image is determined according to the similarity. Specifically, whether the similarity meets a preset condition is detected, and a hand with the similarity meeting the preset condition is used as a target hand, wherein the preset condition is that the similarity is greater than the preset similarity; if the similarity meets the preset condition, the matching degree of the hand features in the currently recognized first image and the hand features of the head-mounted display device wearer is high, then executing gesture recognition operation through the gesture recognition module to recognize the hand motions of the hand of the wearer and acquire the current gesture recognition result of the wearer. The gesture recognition module can be a gesture sensor, an IR LED and a COMS photosensitive array are integrated in the gesture sensor, when the gesture recognition module works normally, the IR LED emits infrared light, the infrared light is reflected after encountering an object, reflected light is received by the high-precision COMS photosensitive array, hand motions are recognized according to the characteristics of the reflected light, and the gesture sensor can recognize various hand motions such as left, right, up, down, front, back, clockwise rotation, anticlockwise rotation, hand waving, hand circling and the like.
Further, the step of determining a target hand belonging to the wearer in the first image according to the similarity comprises:
step S31, determining the hand with the similarity greater than a preset similarity, and taking the hand with the similarity greater than the preset similarity as a target hand, where the target hand is the hand belonging to the wearer in the first image.
In this embodiment, the predetermined condition is greater than the predetermined similarity. Detecting whether the similarity is greater than a preset similarity or not, and taking the hand with the similarity greater than the preset similarity as a target hand; if the similarity meets the preset condition, the matching degree of the hand features in the currently recognized first image and the hand features of the head-mounted display device wearer is high, and then the gesture recognition operation is executed through the gesture recognition module.
According to the gesture recognition method provided by the embodiment, if a head-mounted display device detects a gesture recognition instruction, a first image in a shooting area of the head-mounted display device is collected in real time, and a first hand feature point in the first image is extracted, wherein the first image is an image containing hand features of a user to be recognized; determining a second hand feature point corresponding to the first hand feature point in a feature library of the head-mounted display device, and determining the similarity between the first hand feature point and the second hand feature point; determining target hands belonging to a wearer in the first image according to the similarity, and determining the number of the target hands corresponding to the target hands; and if the number of the target hands is larger than or equal to a first preset value, executing gesture recognition operation corresponding to the gesture recognition instruction. In this embodiment, before the gesture is recognized, a first image is collected, and a first hand feature point in the first image is extracted; comparing the first hand characteristic points with pre-stored second hand characteristic points of the wearer, and calculating the similarity between the first hand characteristic points and the second hand characteristic points to identify whether the currently identified hand is the hand of the wearer; if the similarity between the first hand feature point and the second hand feature point meets a preset condition, which indicates whether the currently recognized hand is the hand of the wearer, and the hand features of the non-wearer in the first image are less, the gesture recognition operation may be performed. This embodiment gathers the current first hand feature of the person of wearing earlier before carrying out the gesture recognition operation and verifies, carries out the gesture recognition operation again to can avoid external factors to include the interference of non-person of wearing's gesture to gesture recognition, promote gesture recognition's accuracy.
Based on the first embodiment, a second embodiment of the gesture recognition method of the present invention is proposed, and referring to fig. 3, in this embodiment, step S20 includes:
step S21 of determining the number of hands included in the first image based on the first hand feature point;
step S22, if the number of hands is less than or equal to a second preset value and greater than a third preset value, determining a second hand feature point corresponding to the first hand feature point in a feature library of the head-mounted display device.
In this embodiment, the first hand feature point includes a distance between a fingertip point and a valley point, a hand contour size, a hand color depth, a joint point, or a hand vein feature, and therefore the number of hands included in the first image is determined from the first hand feature. After the first image is obtained and the first hand feature point in the first image is identified, determining the number of hands in the first image according to the first hand feature point, and detecting whether the number of hands in the first image is smaller than or equal to a second preset value and larger than a third preset value, wherein the second preset value is larger than the third preset value, the value range of the second preset value can be 2-4, and the third preset value is 0. If the number of the hands in the first image is smaller than a third preset value, it is determined that the number of the hands in the first image is too small; if the number of the hands in the first image is larger than a second preset value, the number of the hands in the first image is too large. Therefore, when the number of hands is smaller than or equal to the second preset value and larger than the third preset value, the first hand feature point in the first image is compared with the second hand feature point in the feature library of the head-mounted display device, so as to determine the second hand feature point corresponding to the first hand feature point in the feature library of the head-mounted display device.
Further, after the step of determining the number of hands included in the first image based on the first hand feature point, the method further includes:
step S23, if the number of the hands is larger than the second preset value, outputting prompt information that the current gesture contains the hands of the non-wearer and whether the gesture is recognized, and receiving feedback information corresponding to the prompt information;
step S24, based on the feedback information, executing a gesture recognition operation corresponding to the gesture recognition instruction.
In this embodiment, if the number of hands in the first image is greater than the second preset value, which indicates that the number of hands in the first image is too large and includes the hands of the non-wearer, the head-mounted display device outputs the current image or the current gesture including the prompt information of the hands of the non-wearer, informs the user that there is redundant hand interference recognition currently, and outputs the prompt information whether to continue recognizing the gesture. After the prompt information is input, performing gesture recognition operation according to feedback information fed back by the user, wherein the feedback information is a feedback instruction of the user, and can be a gesture, voice information or touch information.
Further, the step of executing the gesture recognition operation corresponding to the gesture recognition instruction based on the feedback information includes:
step S241, if the feedback information includes a first execution instruction corresponding to a gesture recognition operation, executing the gesture recognition operation;
step S242, if the feedback information includes a second execution instruction corresponding to execution of the gesture recognition operation, executing the step of acquiring the first image in the shooting area of the head-mounted display device in real time and extracting the first hand feature point in the first image.
In this embodiment, if the feedback information is a first execution instruction, the gesture recognition operation is performed, where the first execution instruction is a control instruction corresponding to the gesture recognition operation, and the first execution instruction includes gesture information, voice information, touch information, or the like corresponding to the gesture recognition operation. That is to say, when the number of the hands in the first image is greater than the second preset value and the first image contains more hands, if the first execution instruction fed back by the user is received, the gesture recognition operation is continuously executed.
And if the feedback information is a second execution instruction corresponding to the execution of the gesture recognition operation, acquiring the image again, taking the acquired image as a first image, executing the step of acquiring the first image in the shooting area of the head-mounted display device in real time, and extracting a first hand feature point in the first image. The second execution instruction is a control instruction corresponding to the hand image to be reacquired, and the second execution instruction comprises gesture information, voice information or touch information and the like corresponding to the hand image to be reacquired. That is to say, when the number of the hands in the first image is greater than the second preset value and the first image contains more hands, if the second execution instruction fed back by the user is received, the hand image is collected again to identify the gesture of the user.
Further, before the step of determining a second hand feature point corresponding to the first hand feature point in the feature library of the head-mounted display device and determining the similarity between the first hand feature point and the second hand feature point, the method further includes:
acquiring a second image in a shooting area of the head-mounted display device, and extracting a second hand feature point in the second image, wherein the second image is an image containing the hand feature of the wearer;
storing the second hand feature point in a feature library of the head-mounted display device.
In this embodiment, the second hand feature point pre-stored in the feature library is a plurality of kinds of hand feature information pre-stored by the head-mounted display device wearer. It should be noted that a second image of the wearer is acquired in advance, and the hand features of the wearer in the second image are extracted to obtain second hand feature points; after the second hand characteristic point of the wearer is acquired, the second hand characteristic point is stored in a characteristic library of the head-mounted display device for the hand of the wearer to be compared based on the second hand characteristic point of the wearer stored in the characteristic library in the gesture recognition process.
Further, the second image comprises one or more of a hand front unfolded image, a left unfolded image, a right unfolded image, a back unfolded image, a fist front image, a fist side image, or a fist back image, and the second hand feature points comprise one or more of a distance between a fingertip point and a valley point, a hand contour size, a hand color depth, an articulation point, or a hand vein feature.
Further, after the step of executing the gesture recognition operation corresponding to the gesture recognition instruction, the method further includes:
acquiring a gesture recognition result based on the gesture recognition operation, and determining an operation instruction corresponding to the gesture recognition result;
executing the operation instruction on the head-mounted display device.
In this embodiment, the gesture recognition module performs a gesture recognition operation to recognize a hand motion of the hand of the wearer, and obtains a current gesture recognition result of the wearer. After the gesture recognition result is recognized, an operation instruction corresponding to the gesture recognition result is determined according to the gesture recognition result, and therefore target operation corresponding to the operation instruction is executed on the head-mounted display device. The operation instruction is a control instruction for controlling the head-mounted display device, and the operation instruction may be a left-sliding instruction, a determining instruction, a quitting instruction, and the like, and the operation instruction is not specifically limited in this embodiment.
In the gesture recognition method provided in this embodiment, the number of hands included in the first image is determined based on the first hand feature point; and if the number of the hands is smaller than or equal to a second preset numerical value and larger than a third preset numerical value, determining a second hand characteristic point corresponding to the first hand characteristic point in a characteristic library of the head-mounted display device. In the embodiment, the number of the hands in the first image is identified to be less than or equal to the second preset value and greater than the third preset value, and then the gesture identification operation is executed, so that the interference of external factors including the gesture of a non-wearer on the gesture identification can be avoided, and the accuracy of the gesture identification is improved.
In addition, an embodiment of the present invention further provides a nonvolatile storage medium, where a gesture recognition program is stored on the nonvolatile storage medium, and the gesture recognition program, when executed by a processor, implements the steps of the gesture recognition method according to any one of the above.
The specific embodiment of the non-volatile storage medium of the present invention is substantially the same as the embodiments of the gesture recognition method described above, and will not be described in detail herein.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A gesture recognition method is applied to a head-mounted display device and is characterized by comprising the following steps:
if the head-mounted display equipment detects a gesture recognition instruction, acquiring a first image in a shooting area of the head-mounted display equipment in real time, and extracting a first hand feature point in the first image, wherein the first image is an image containing hand features of a user to be recognized;
determining a second hand feature point corresponding to the first hand feature point in a feature library of the head-mounted display device, and determining the similarity between the first hand feature point and the second hand feature point;
determining target hands belonging to a wearer in the first image according to the similarity, and determining the number of the target hands corresponding to the target hands;
and if the number of the target hands is larger than or equal to a first preset value, executing gesture recognition operation corresponding to the gesture recognition instruction.
2. The gesture recognition method of claim 1, wherein the step of determining a second hand feature point corresponding to the first hand feature point in a feature library of the head mounted display device comprises:
determining the number of hands contained in the first image based on the first hand feature point;
and if the number of the hands is smaller than or equal to a second preset numerical value and larger than a third preset numerical value, determining a second hand characteristic point corresponding to the first hand characteristic point in a characteristic library of the head-mounted display device.
3. The gesture recognition method according to claim 2, wherein after the step of determining the number of hands included in the first image based on the first hand feature point, the method further comprises:
if the number of the hands is larger than the second preset value, outputting prompt information that the current gesture contains the hands of the non-wearer and whether the gesture is recognized or not, and receiving feedback information corresponding to the prompt information;
and executing gesture recognition operation corresponding to the gesture recognition instruction based on the feedback information.
4. The gesture recognition method according to claim 3, wherein the step of executing the gesture recognition operation corresponding to the gesture recognition instruction based on the feedback information comprises:
if the feedback information contains a first execution instruction corresponding to the gesture recognition operation, executing the gesture recognition operation;
and if the feedback information contains a second execution instruction corresponding to the execution of the gesture recognition operation, executing the step of acquiring a first image in a shooting area of the head-mounted display equipment in real time and extracting a first hand feature point in the first image.
5. The gesture recognition method of claim 1, wherein the step of determining the target hand belonging to the wearer in the first image according to the similarity comprises:
determining the hand with the similarity larger than the preset similarity, and taking the hand with the similarity larger than the preset similarity as a target hand, wherein the target hand is the hand belonging to the wearer in the first image.
6. The gesture recognition method according to claim 1, wherein the step of determining a second hand feature point corresponding to the first hand feature point in the feature library of the head-mounted display device and determining the similarity between the first hand feature point and the second hand feature point further comprises:
acquiring a second image in a shooting area of the head-mounted display device, and extracting a second hand feature point in the second image, wherein the second image is an image containing the hand feature of the wearer;
storing the second hand feature point in a feature library of the head-mounted display device.
7. The gesture recognition method of claim 6, wherein the second image comprises one or more of a hand frontal unfolded image, a left unfolded image, a right unfolded image, a back unfolded image, a fist frontal image, a fist side image, or a fist back image, and the second hand feature points comprise one or more of a distance between a fingertip point and a valley point, a hand contour size, a hand color depth, a joint point, or a hand vein feature.
8. The gesture recognition method according to any one of claims 1 to 7, wherein after the step of performing the gesture recognition operation corresponding to the gesture recognition instruction, the method further comprises:
acquiring a gesture recognition result based on the gesture recognition operation, and determining an operation instruction corresponding to the gesture recognition result;
executing the operation instruction on the head-mounted display device.
9. A head-mounted display device, comprising: memory, a processor and a gesture recognition program stored on the memory and executable on the processor, the gesture recognition program when executed by the processor implementing the steps of the gesture recognition method according to any one of claims 1 to 8.
10. A non-volatile storage medium having stored thereon a gesture recognition program which, when executed by a processor, implements the steps of the gesture recognition method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110364942.8A CN113158845A (en) | 2021-04-02 | 2021-04-02 | Gesture recognition method, head-mounted display device and nonvolatile storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110364942.8A CN113158845A (en) | 2021-04-02 | 2021-04-02 | Gesture recognition method, head-mounted display device and nonvolatile storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113158845A true CN113158845A (en) | 2021-07-23 |
Family
ID=76888738
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110364942.8A Pending CN113158845A (en) | 2021-04-02 | 2021-04-02 | Gesture recognition method, head-mounted display device and nonvolatile storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113158845A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140153774A1 (en) * | 2012-12-04 | 2014-06-05 | Alpine Electronics, Inc. | Gesture recognition apparatus, gesture recognition method, and recording medium |
| CN111273777A (en) * | 2020-02-11 | 2020-06-12 | Oppo广东移动通信有限公司 | Virtual content control method and device, electronic equipment and storage medium |
| CN111310608A (en) * | 2020-01-22 | 2020-06-19 | Oppo广东移动通信有限公司 | User identification method, user identification device, storage medium and head-mounted device |
| CN111930226A (en) * | 2020-07-01 | 2020-11-13 | 青岛小鸟看看科技有限公司 | Gesture tracking method and device |
| CN112215167A (en) * | 2020-10-14 | 2021-01-12 | 上海爱购智能科技有限公司 | Intelligent store control method and system based on image recognition |
| CN112362077A (en) * | 2020-11-13 | 2021-02-12 | 歌尔光学科技有限公司 | Head-mounted display device, obstacle avoidance method thereof and computer-readable storage medium |
-
2021
- 2021-04-02 CN CN202110364942.8A patent/CN113158845A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140153774A1 (en) * | 2012-12-04 | 2014-06-05 | Alpine Electronics, Inc. | Gesture recognition apparatus, gesture recognition method, and recording medium |
| CN111310608A (en) * | 2020-01-22 | 2020-06-19 | Oppo广东移动通信有限公司 | User identification method, user identification device, storage medium and head-mounted device |
| CN111273777A (en) * | 2020-02-11 | 2020-06-12 | Oppo广东移动通信有限公司 | Virtual content control method and device, electronic equipment and storage medium |
| CN111930226A (en) * | 2020-07-01 | 2020-11-13 | 青岛小鸟看看科技有限公司 | Gesture tracking method and device |
| CN112215167A (en) * | 2020-10-14 | 2021-01-12 | 上海爱购智能科技有限公司 | Intelligent store control method and system based on image recognition |
| CN112362077A (en) * | 2020-11-13 | 2021-02-12 | 歌尔光学科技有限公司 | Head-mounted display device, obstacle avoidance method thereof and computer-readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220326781A1 (en) | Bimanual interactions between mapped hand regions for controlling virtual and graphical elements | |
| US9207771B2 (en) | Gesture based user interface | |
| US10585488B2 (en) | System, method, and apparatus for man-machine interaction | |
| KR20220062338A (en) | Hand pose estimation from stereo cameras | |
| CN108525305B (en) | Image processing method, image processing device, storage medium and electronic equipment | |
| KR20200101206A (en) | Method for providing shoot mode based on virtual character and electronic device performing thereof | |
| US20140240225A1 (en) | Method for touchless control of a device | |
| EP3671549B1 (en) | Electronic device for assisting a user during exercise | |
| CN108989678B (en) | Image processing method and mobile terminal | |
| US11599070B2 (en) | Electronic device and method for determining task including plural actions | |
| CN107273869B (en) | Gesture recognition control method and electronic equipment | |
| KR20200028771A (en) | Electronic device and method for recognizing user gestures based on user intention | |
| CN112613475A (en) | Code scanning interface display method and device, mobile terminal and storage medium | |
| JP2021531589A (en) | Motion recognition method, device and electronic device for target | |
| US11335090B2 (en) | Electronic device and method for providing function by using corneal image in electronic device | |
| CN111736709A (en) | AR glasses control method, device, storage medium and apparatus | |
| CN112540696A (en) | Screen touch control management method, intelligent terminal, device and readable storage medium | |
| US20230074380A1 (en) | Electronic device for providing augmented reality service and operating method thereof | |
| US11029753B2 (en) | Human computer interaction system and human computer interaction method | |
| CN113158845A (en) | Gesture recognition method, head-mounted display device and nonvolatile storage medium | |
| CN114510142B (en) | Gesture recognition method based on two-dimensional image, gesture recognition system based on two-dimensional image and electronic equipment | |
| CN114546103A (en) | Operation method through gestures in augmented reality and head-mounted display system | |
| US11054941B2 (en) | Information processing system, information processing method, and program for correcting operation direction and operation amount | |
| CN110852221A (en) | Intelligent face recognition method based on block combination, terminal and storage medium | |
| CN111919222B (en) | Apparatus and method for recognizing object in image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20221125 Address after: 266104 No. 500, Songling Road, Laoshan District, Qingdao, Shandong Applicant after: GOERTEK TECHNOLOGY Co.,Ltd. Address before: 261031 workshop 1, phase III, Geer Photoelectric Industrial Park, 3999 Huixian Road, Yongchun community, Qingchi street, high tech Zone, Weifang City, Shandong Province Applicant before: GoerTek Optical Technology Co.,Ltd. |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210723 |