CN116166120A

CN116166120A - Gesture recognition method and device, electronic equipment and storage medium

Info

Publication number: CN116166120A
Application number: CN202310067471.3A
Authority: CN
Inventors: 汪铭扬
Original assignee: Vivo Mobile Communication Co Ltd
Current assignee: Vivo Mobile Communication Co Ltd
Priority date: 2023-01-12
Filing date: 2023-01-12
Publication date: 2023-05-26

Abstract

The application discloses a gesture recognition method, a gesture recognition device, electronic equipment and a storage medium, and belongs to the field of gesture recognition. Applied to a head-mounted display device, the method comprises: acquiring an image of a user's hand through the head-mounted display device; acquiring pose data of a first hand of a user through first equipment, wherein the first hand is a part of the hand of the user which is covered under the view angle of the head-mounted display equipment; and determining the gesture of the user according to the image and the pose data.

Description

Gesture recognition method and device, electronic equipment and storage medium

Technical Field

The application belongs to the field of gesture recognition, and particularly relates to a gesture recognition method, a gesture recognition device, electronic equipment and a storage medium.

Background

Currently, with the continuous development of technology, more and more various electronic products, such as smart glasses, wear display devices and the like, and enter the lives of consumers, and consumers are rapidly adapting to the products, so that the smart glasses are gradually applied to various aspects of daily life. When a user controls the head-mounted display device, gestures are generally used for interacting with the head-mounted display device, the head-mounted display device obtains gesture images through a camera of the head-mounted display device, and the gestures of the user are determined.

However, the gesture recognition of the existing head-mounted display device is difficult to accurately and timely track and determine the gesture of the user, and at certain angles, the hand of the user can be blocked by other objects or the hand of the user, and at the moment, the gesture of the user cannot be recognized only through the head-mounted display device.

Disclosure of Invention

The embodiment of the application aims to provide a gesture recognition method, device, electronic equipment and storage medium, which can solve the problem that a head-mounted display device is difficult to accurately determine gestures of a user.

In a first aspect, an embodiment of the present application provides a method for gesture recognition, applied to a head-mounted display device, where the method includes: acquiring an image of a user's hand through the head-mounted display device; acquiring pose data of a first hand of a user through first equipment, wherein the first hand is a part of the hand of the user which is covered under the view angle of the head-mounted display equipment; and determining the gesture of the user according to the image and the pose data.

Optionally, the acquiring, by the first device, pose data of the first hand of the user includes: when the first device comprises a wearable device, determining pose data of the first hand according to the motion data of the user hand, wherein the motion data of the user hand is detected by a motion detection unit in the wearable device; detecting a millimeter wave signal by a millimeter wave detection unit in a hand-operated device in a case where the first device includes the hand-operated device; determining pose data of a first hand corresponding to the millimeter wave signal according to the millimeter wave signal; in the case that the first device comprises a spatial positioning device, acquiring an image of a user's hand or measuring distances of a plurality of target positions of the user's hand by an image acquisition device or a distance measurement device in the spatial positioning, and determining pose data of the first hand according to the image or the distances.

Optionally, the first device includes a wearing device worn on a hand of the user, the image includes an image of the hand of the user, an image of the wearing device, and an image displayed by the wearing device, and determining, according to the image and the pose data, a gesture of the user includes: identifying characteristic points from the image, wherein the characteristic points comprise joint points of the hand of the user, characteristic points of the wearing equipment and characteristic points of the image displayed by the wearing equipment; and determining the gesture of the user according to the position of the characteristic point and the pose data of the first hand.

Optionally, the first device comprises a hand manipulation device; the hand control equipment is provided with a touch detection unit; the acquiring, by the head-mounted display device, an image of a hand of a user includes: and under the condition that the hand of the user does not contact the touch detection area, acquiring an image of the hand of the user through the head-mounted display device.

Optionally, before the pose data of the first hand of the user is acquired by the first device, the method further includes: acquiring position information and parameter information of a plurality of space positioning devices around the head-mounted display device; the position information includes a distance and an angle of a spatial positioning device relative to the head mounted display device; and determining the first equipment from a plurality of space positioning equipment according to the position information and the parameter information.

In a second aspect, an embodiment of the present application provides an apparatus for gesture recognition, applied to a head-mounted display device, where the apparatus includes: the first acquisition module is used for acquiring an image of the hand of the user through the head-mounted display device; the second acquisition module acquires pose data of a first hand of a user through first equipment, wherein the first hand is a part of the hand of the user which is covered under the view angle of the head-mounted display equipment; and the first determining module is used for determining gestures of a user according to the images and the pose data.

Optionally, the second obtaining module includes: a first determining submodule, configured to determine pose data of a first hand according to motion data of the user hand detected by a motion detection unit in a wearable device, where the first device includes the wearable device; a second determination sub-module configured to detect a millimeter wave signal by a millimeter wave detection unit in a hand-manipulation device in a case where the first device includes the hand-manipulation device; determining pose data of a first hand corresponding to the millimeter wave signal according to the millimeter wave signal; and the third determining submodule is used for acquiring images of the hands of the user or measuring the distances of a plurality of target positions of the hands of the user through the image acquisition equipment or the distance measuring equipment in the space positioning under the condition that the first equipment comprises the space positioning equipment, and determining the pose data of the first hand according to the images or the distances.

Optionally, the first device includes a wearing device worn on a hand of the user, the image includes an image of the hand of the user, an image of the wearing device, and an image displayed by the wearing device, and the first determining module includes: the identification sub-module is used for identifying characteristic points from the image, wherein the characteristic points comprise joint points of the hand of the user, characteristic points of the wearing equipment and characteristic points of the image displayed by the wearing equipment; and the fourth determining submodule is used for determining gestures of the user according to the positions of the characteristic points and the pose data of the first hand.

Optionally, the first device comprises a hand manipulation device; the hand control equipment is provided with a touch detection unit; the first acquiring module is specifically configured to acquire, through the head-mounted display device, an image of a hand of the user when the hand of the user does not contact the touch detection area.

Optionally, the apparatus further comprises: the third acquisition module is used for acquiring position information and parameter information of a plurality of space positioning devices around the head-mounted display device before the pose data of the first hand of the user are acquired through the first device; the position information includes a distance and an angle of a spatial positioning device relative to the head mounted display device; and the second determining module is used for determining the first equipment in the plurality of space positioning equipment according to the position information and the parameter information.

In a third aspect, embodiments of the present application provide an electronic device comprising a processor, a memory and a program or instruction stored on the memory and executable on the processor, the program or instruction implementing the steps of the method according to the first aspect when executed by the processor.

In a fourth aspect, embodiments of the present application provide a readable storage medium having stored thereon a program or instructions which when executed by a processor implement the steps of the method according to the first aspect.

In a fifth aspect, embodiments of the present application provide a chip, where the chip includes a processor and a communication interface, where the communication interface is coupled to the processor, and where the processor is configured to execute a program or instructions to implement a method according to the first aspect.

In a sixth aspect, embodiments of the present application provide a computer program product stored in a storage medium, the program product being executable by at least one processor to implement the method according to the first aspect.

In the embodiment of the application, the gesture of the user can be determined according to the image of the hand of the user and the pose data of the hand of the user, which are obtained by the head-mounted display device, and the pose data of the hand of the user, which are covered under the view angle of the head-mounted display device, are obtained by the first device. By the method, when hands in the images acquired by the head-mounted display device are shielded, gesture data of the shielded hands can be acquired according to other devices, gestures of a user are judged, accuracy of gesture determination is improved, and the problem of gesture recognition when the hands are shielded from the view angle of the head-mounted display device is solved.

Drawings

FIG. 1 is a flow chart of a method of gesture recognition provided by an embodiment of the present application;

FIG. 2 is a schematic structural diagram of a gesture recognition apparatus according to an embodiment of the present application;

fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of an electronic device according to another embodiment of the present application.

Detailed Description

Technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application are within the scope of the protection of the present application.

The terms first, second and the like in the description and in the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged, as appropriate, such that embodiments of the present application may be implemented in sequences other than those illustrated or described herein, and that the objects identified by "first," "second," etc. are generally of a type and not limited to the number of objects, e.g., the first object may be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/", generally means that the associated object is an "or" relationship.

The gesture recognition method provided by the embodiment of the application is described in detail below through specific embodiments and application scenes thereof with reference to the accompanying drawings.

Referring to fig. 1, a flowchart of a gesture recognition method according to an embodiment of the present application is provided. The method can be applied to a head-mounted display device, and particularly the head-mounted display device can be VR (virtual reality), AR (augmented reality) intelligent glasses and the like. In one example, the head mounted display device may be communicatively coupled to the first device, and in particular may be communicatively coupled thereto via bluetooth, WIFI, UWB, or the like. As shown in fig. 1, the method may include steps S11 to S13, which will be described in detail below.

Step S11, acquiring an image of the hand of the user through the head-mounted display device.

In step S12, pose data of a first hand of the user is acquired by the first device, where the first hand is a portion of the hand of the user that is covered under the view angle of the head-mounted display device.

In an example of this embodiment, the head-mounted display device may capture an image of a user's hand through its own camera, where the image of the user's hand may include a picture and an image of the user's hand, and in an example, the head-mounted display device is further provided with a binocular camera, and when acquiring the image of the user's hand, depth data of the user's hand may also be acquired simultaneously to assist in determining the gesture of the user's hand.

In an example of this embodiment, when the head-mounted display device captures an image of a user's hand, the user's hand may be blocked by another object or the hand itself due to a problem of its own viewing angle, at this time, the head-mounted display device may determine, according to the image of the user's hand, a portion of the user's hand that is blocked, that is, a portion of the first hand, and specifically, for example, may determine, by identifying a joint point of the user's hand in the image, which portions of the user's hand are portions that are not blocked, and which portions are portions that are blocked.

In one example of this embodiment, after the first device obtains the pose data of the first hand of the user, the obtained pose data of the hand of the user may be sent to the head-mounted display device through the communication connection between the first device and the head-mounted display device.

In one example of this embodiment, the first device may include at least one of the following: the wearing equipment is worn on the hand of the user; a hand manipulation device; the space positioning device comprises an image acquisition device or a distance measuring device.

In one example of this embodiment, the first device may be one or more, and in particular, the first device may be a wearable device worn on a user's hand, such as a smart watch, smart bracelet, or smart ring, among others. The first device may also be a hand manipulation device, such as a handle or the like. The first device may be provided therein with a millimeter wave detection unit. The first device may also be a spatial positioning device, which is a device that may assist the head-mounted display device in spatially positioning the hand, and the spatial positioning device may include an image acquisition device and a ranging device. The image acquisition device is a device with image acquisition capability, such as a smart television with a camera, a smart refrigerator, a monitoring camera, a mobile phone and the like. The ranging device is a device with a ranging function, for example, a device with functions of LiDar (Laser Radar), UWB, millimeter wave, and the like.

In one example of this embodiment, obtaining, by the first device, pose data of a first hand of a user includes: in the case that the first device includes a wearable device, the pose data of the first hand is determined according to the motion data of the user hand by the motion data of the user hand detected by the motion detection unit in the wearable device.

In an example of this embodiment, a motion detection unit may also be provided in the wearable device, specifically, for example, a gyroscope, an accelerometer, an inertial measurement unit, and the like, and the wearable device may acquire motion data of the hand of the user, including data of a rotation angle, a displacement distance, and the like, through the motion detection unit. After the motion data of the hand of the user is determined, pose data of the hand of the user can be determined according to the motion data of the hand of the user, wherein the pose data of the hand of the user comprise pose data of the part of the hand which is shielded.

In one example of the present embodiment, in the case where the first device includes a hand-manipulation device, the millimeter wave signal is detected by a millimeter wave detection unit in the hand-manipulation device; determining pose data of a first hand corresponding to the millimeter wave signal according to the millimeter wave signal;

In one example of the present embodiment, the millimeter wave detection unit in the hand manipulation device may transmit a set millimeter wave signal to the periphery of the device, receive the reflected millimeter wave signal, and recognize the waveform of the millimeter wave signal, and determine the position and posture of the hand of the user based on the waveform. In one example, waveforms of all reflected millimeter wave signals of gestures that can interact with the head-mounted display device may be stored in advance, and after the millimeter wave signals reflected by the hands of the user are received, the waveforms of the signals may be compared with the pre-stored waveforms to determine pose data of the hands of the user. The pose data of the hand of the user comprise pose data of the part of the hand which is shielded. Further, after the pose data of the first hand is determined, the data may be transmitted to the head mounted display device. The head-mounted display device can accurately determine the gesture of the user according to the pose data and the image shot by the head-mounted display device.

In one example of the present embodiment, in the case where the first device includes a spatial positioning device, an image of the user's hand is acquired or distances of a plurality of target positions of the user's hand are measured by an image acquisition device or a distance measurement device in spatial positioning, and pose data of the first hand is determined from the image or the distances.

In an example of this embodiment, in a case where the first device includes a unified positioning device, the positioning device may be controlled to collect images of the hand of the user or detect distances between points of the hand of the user from other angles, and determine pose data of the hand of the user according to the images of the other angles or the distances between points of the hand of the user, where the pose data of the hand of the user includes pose data of a portion of the hand that is blocked. After obtaining the pose data of the first hand of the user, the pose data of the first hand may be sent to the head mounted display device based on the communication connection with the head mounted display device.

In the embodiment, in the process of determining the gesture of the user, the head-mounted display device can assist the head-mounted display device to acquire the pose of the blocked part of the hand through various first devices and through various modes such as images, millimeter waves, ranging and motion data, so that the accuracy of gesture determination is improved, and the problem of gesture recognition when the hand is blocked in the view angle of the head-mounted display device is solved.

In one example of the present embodiment, where the first device includes a spatial positioning device, obtaining, by the first device, user hand pose data includes: the method comprises the steps of sending first request information to space positioning equipment to trigger the space positioning equipment to collect pose data of a hand of a user; and receiving pose data sent by the space positioning equipment.

In one example of this embodiment, when the spatial positioning device is required to collect pose data of the user's hand, the head-mounted display device may send request information to the device based on the communication connection, so that the spatial positioning device collects pose data of the user's hand and sends the pose data to the head-mounted display device.

Step S13, determining gestures of a user according to the images and the pose data.

In one example of this embodiment, after acquiring the image of the user's hand and the pose data of the first hand, the smart wearable device may determine the pose of the hand of the non-occluded portion according to the joint points of the user's hand in the image, and determine the gesture of the user in combination with the pose of the hand of the occluded portion.

In this example, the gesture of the user may be determined according to the image of the hand of the user and the pose data of the hand of the user, which is covered under the view angle of the head-mounted display device, by acquiring the image of the hand of the user through the head-mounted display device, and acquiring the pose data of the hand of the user through the first device. By the method, when hands in the images acquired by the head-mounted display device are shielded, gesture data of the shielded hands can be acquired according to other devices, gestures of a user are judged, accuracy of gesture determination is improved, and the problem of gesture recognition when the hands are shielded from the view angle of the head-mounted display device is solved.

In one example of this embodiment, the first device includes a wearable device worn on a hand of a user, the image includes an image of the hand of the user, an image of the wearable device, and an image displayed by the wearable device, and determining a gesture of the user based on the image and the pose data includes: identifying characteristic points from the image, wherein the characteristic points comprise joint points of the hand of the user, characteristic points of wearing equipment and characteristic points of the image displayed by the wearing equipment; and determining the gesture of the user according to the position of the characteristic point and the pose data of the first hand.

In an example of this embodiment, the images of the hand of the user acquired by the head-mounted display device may include an image of a portion of the hand of the user taken from one image, an image of a portion of the wearing device, and an image of a portion displayed by the wearing device, or may be images of three portions respectively acquired.

In one example of the present embodiment, the feature points include an articulation point of a user's hand, a feature point of an image displayed by the wearable device, and a feature point of the wearable device. The articulation points of the hand may be points in the image where the wrist and finger joints are located. The feature point of the wearable device may be some feature point of itself, taking the smart watch as an example, and the feature point may be some point at the watch band or at the dial of the smart watch. In one example, the wearable device may also display a specific image on its screen when the auxiliary head-mounted display device confirms the gesture, so that the head-mounted display device may confirm the user gesture according to the feature point of the specific image.

In an example of this embodiment, in the process of making a gesture by the hand of the user, the head-mounted display device may determine a distance change of each position of the hand according to a position change of a feature point in the image, so as to determine a pose change of the hand of the user, and may also accurately determine the gesture of the user by combining pose data of the first hand.

In one example of the present embodiment, the first device comprises a hand manipulation device; the hand control device is provided with a touch detection unit; acquiring, by a head-mounted display device, an image of a user's hand, comprising: in the case where the user's hand does not contact the touch detection area, an image of the user's hand is acquired by the head-mounted display device.

In one example of the present embodiment, a hand manipulation device such as a handle or the like may be provided with a touch detection unit such as a capacitive touch detection unit to detect whether or not a hand is in contact with a touch detection area, which may be set as an area where the hand is in contact with the handle when the user normally holds the handle, and the touch detection unit may detect that the hand is in contact with the handle when the user normally holds the handle. The user's hand is not in contact with the touch detection area, which may include a case where there is no contact in part or a case where there is no contact in whole, and the user's hand may be required to interact with the head-mounted display device by using a gesture when the hand-mounted display device acquires an image of the user's hand and the gesture of the hand acquired by the first device to determine the gesture of the user.

In one example of this embodiment, before the pose data of the first hand of the user is acquired by the first device, the method further comprises: acquiring position information and parameter information of a plurality of space positioning devices around the head-mounted display device; the position information includes a distance and an angle of the spatial locating device relative to the head mounted display device; and determining the first equipment from the plurality of space positioning equipment according to the position information and the parameter information.

In one example of this embodiment, the head mounted display device may communicate with a plurality of spatial locating devices around the device in advance based on the communication connection to obtain the relative distances and angles of the plurality of spatial locating devices around and the head mounted display device. For example, the relative distance and angle may be determined by accepting a particular signal transmitted by the spatial locating device and based on the particular signal. Meanwhile, parameter information of the spatial positioning device can also be obtained, and specifically, for example, when the spatial positioning device is an image acquisition device, the parameter information can include parameters of pixels, shooting capability, camera type and the like of a camera of the image acquisition device. In the case where the spatial locating device is a ranging device, parameters representing its locating capability, such as the measuring range, measuring accuracy, and measuring granularity of the device, may be included.

In one example of this embodiment, after the location information and the parameter information of the plurality of spatial positioning devices are acquired, a device most suitable for assisting in determining the gesture of the user may be determined according to the information. Specifically, for example, the device capable of acquiring the first hand pose of the user from among the plurality of spatial positioning devices may be determined first according to the parameter information and the position information, and one nearest device is selected as the first device from among the devices capable of acquiring the first hand pose of the user. The first device may then obtain the user hand pose data, which may specifically include an image of the user's hand at a different angle than the head-mounted display device or a change in distance and pose of the user's hand. The head-mounted display device can determine the gesture of the user according to the pose data and the hand image shot by the head-mounted display device.

It should be noted that, although the above examples illustrate a specific manner of determining the first device according to the location information and the parameter information, those skilled in the art can understand that the present disclosure is not limited thereto, and how to determine the first device according to the location information and the parameter information specifically, and those skilled in the art can flexibly set according to actual needs.

In this example, under the condition that a plurality of space positioning devices are arranged around the head-mounted display device, the device most suitable for assisting the head-mounted display device in determining the gesture of the user can be determined according to the position information and the parameter information of the plurality of space positioning devices, the most suitable device is selected on the basis of ensuring that the gesture can be identified, and the problem of resource waste caused by using the plurality of space positioning devices is avoided.

According to the gesture recognition method provided by the embodiment of the application, the execution subject can be a gesture recognition device. In the embodiments of the present application, a method for performing gesture recognition by using a gesture recognition device is taken as an example, and the gesture recognition device provided in the embodiments of the present application is described.

Corresponding to the above embodiments, referring to fig. 2, the embodiment of the present application further provides a gesture recognition apparatus 100, including: a first acquiring module 101, configured to acquire an image of a hand of a user through a head-mounted display device; the second obtaining module 102 obtains pose data of a first hand of a user through the first device, wherein the first hand is a part of the hand of the user which is covered under the view angle of the head-mounted display device; the first determining module 103 is configured to determine a gesture of a user according to the image and the pose data.

Optionally, the second acquisition module includes: the first determining submodule is used for determining pose data of the first hand according to the motion data of the hand of the user, which is detected by the motion detection unit in the wearing equipment, when the first equipment comprises the wearing equipment; a second determination sub-module for detecting a millimeter wave signal by a millimeter wave detection unit in the hand manipulation device in a case where the first device includes the hand manipulation device; determining pose data of a first hand corresponding to the millimeter wave signal according to the millimeter wave signal; and the third determining submodule is used for acquiring images of the hands of the user or measuring the distances of a plurality of target positions of the hands of the user through an image acquisition device or a distance measuring device in spatial positioning under the condition that the first device comprises the spatial positioning device, and determining pose data of the first hand according to the images or the distances.

Optionally, the first device includes a wearing device worn on a hand of the user, the image includes an image of the hand of the user, an image of the wearing device, and an image displayed by the wearing device, and the first determining module includes: the identification sub-module is used for identifying characteristic points from the image, wherein the characteristic points comprise joint points of the hand of the user, characteristic points of wearing equipment and characteristic points of the image displayed by the wearing equipment; and the fourth determining submodule is used for determining gestures of the user according to the positions of the feature points and the pose data of the first hand.

Optionally, the first device comprises a hand manipulation device; the hand control device is provided with a touch detection unit; the first acquiring module is specifically configured to acquire an image of a user's hand through the head-mounted display device when the user's hand does not contact the touch detection area.

Optionally, the apparatus further comprises: the third acquisition module is used for acquiring position information and parameter information of a plurality of space positioning devices around the head-mounted display device before the pose data of the first hand of the user are acquired through the first device; the position information includes a distance and an angle of the spatial locating device relative to the head mounted display device; and the second determining module is used for determining the first equipment in the plurality of space positioning equipment according to the position information and the parameter information.

In this example, an apparatus is provided, which may obtain, through a head-mounted display device, an image of a hand of a user, and obtain, through a first device, pose data of the hand of the user that is hidden under a viewing angle of the head-mounted display device, and determine a gesture of the user according to the image of the hand of the user and the pose data of the hand. By the method, when hands in the images acquired by the head-mounted display device are shielded, gesture data of the shielded hands can be acquired according to other devices, gestures of a user are judged, accuracy of gesture determination is improved, and the problem of gesture recognition when the hands are shielded from the view angle of the head-mounted display device is solved.

The gesture recognition apparatus in the embodiments of the present application may be an electronic device, or may be a component in an electronic device, for example, an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices than a terminal. By way of example, the electronic device may be a mobile phone, tablet computer, notebook computer, palm computer, vehicle-mounted electronic device, mobile internet appliance (Mobile Internet Device, MID), augmented reality (augmented reality, AR)/Virtual Reality (VR) device, robot, wearable device, ultra-mobile personal computer, UMPC, netbook or personal digital assistant (personal digital assistant, PDA), etc., but may also be a server, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (TV), teller machine or self-service machine, etc., and the embodiments of the present application are not limited in particular.

The gesture recognition device in the embodiment of the present application may be a device with an operating system. The operating system may be an Android operating system, an ios operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

The gesture recognition device provided in the embodiment of the present application can implement each process implemented by the above method embodiment, and in order to avoid repetition, details are not repeated here.

In correspondence to the above embodiment, optionally, as shown in fig. 3, the embodiment of the present application further provides an electronic device 800, including a processor 801 and a memory 802, where a program or an instruction that can be executed on the processor 801 is stored in the memory 802, and the program or the instruction when executed by the processor 801 implements each step of the above gesture recognition method embodiment, and the same technical effect can be achieved, so that repetition is avoided, and no further description is given here.

The electronic device in the embodiment of the application includes the mobile electronic device and the non-mobile electronic device described above.

Fig. 4 is a schematic hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 900 includes, but is not limited to: radio frequency unit 901, network module 902, audio output unit 903, input unit 904, sensor 905, display unit 906, user input unit 907, interface unit 908, memory 909, and processor 910.

Those skilled in the art will appreciate that the electronic device 900 may also include a power source (e.g., a battery) for powering the various components, which may be logically connected to the processor 910 by a power management system to perform functions such as managing charge, discharge, and power consumption by the power management system. The electronic device structure shown in fig. 4 does not constitute a limitation of the electronic device, and the electronic device may include more or less components than shown, or may combine certain components, or may be arranged in different components, which are not described in detail herein.

Wherein the processor 910 is configured to acquire an image of a hand of a user through the head-mounted display device; acquiring pose data of a first hand of a user through first equipment, wherein the first hand is a part of the hand of the user covered under the view angle of the head-mounted display equipment; and determining the gesture of the user according to the image and the pose data.

Optionally, acquiring, by the first device, pose data of the first hand of the user includes: when the first device comprises a wearable device, determining pose data of the first hand according to motion data of the user hand, wherein the motion data of the user hand is detected by a motion detection unit in the wearable device; detecting a millimeter wave signal by a millimeter wave detection unit in the hand manipulation device in a case where the first device includes the hand manipulation device; determining pose data of a first hand corresponding to the millimeter wave signal according to the millimeter wave signal; in the case where the first device includes a spatial positioning device, an image of the user's hand is acquired or distances of a plurality of target positions of the user's hand are measured by an image acquisition device or a distance measurement device in spatial positioning, and pose data of the first hand is determined from the image or the distances.

Optionally, the first device includes a wearing device worn on a hand of the user, the image includes an image of the hand of the user, an image of the wearing device, and an image displayed by the wearing device, and determining, according to the image and the pose data, a gesture of the user includes: identifying characteristic points from the image, wherein the characteristic points comprise joint points of the hand of the user, characteristic points of wearing equipment and characteristic points of the image displayed by the wearing equipment; and determining the gesture of the user according to the position of the characteristic point and the pose data of the first hand.

Optionally, the first device comprises a hand manipulation device; the hand control device is provided with a touch detection unit; acquiring, by a head-mounted display device, an image of a user's hand, comprising: in the case where the user's hand does not contact the touch detection area, an image of the user's hand is acquired by the head-mounted display device.

Optionally, before acquiring pose data of a first hand of a user by the first device, the processor 910 is configured to acquire position information and parameter information of a plurality of spatial positioning devices around the head-mounted display device; the position information includes a distance and an angle of the spatial locating device relative to the head mounted display device; and determining the first equipment from the plurality of space positioning equipment according to the position information and the parameter information.

In this example, an electronic device is provided, where an image of a hand of a user may be acquired by a head-mounted display device, pose data of the hand of the user that is covered under a viewing angle of the head-mounted display device may be acquired by a first device, and a gesture of the user may be determined according to the image of the hand of the user and the pose data of the hand. By the method, when hands in the images acquired by the head-mounted display device are shielded, gesture data of the shielded hands can be acquired according to other devices, gestures of a user are judged, accuracy of gesture determination is improved, and the problem of gesture recognition when the hands are shielded from the view angle of the head-mounted display device is solved.

It should be appreciated that in embodiments of the present application, the input unit 904 may include a graphics processor (Graphics Processing Unit, GPU) 9041 and a microphone 9042, with the graphics processor 9041 processing image data of still pictures or video obtained by an image capture device (e.g., a camera) in a video capture mode or an image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 907 includes a touch panel 9071 and other input devices 9072. Touch panel 9071, also referred to as a touch screen. The touch panel 9071 may include two parts, a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and so forth, which are not described in detail herein.

The memory 909 may be used to store software programs as well as various data. The memory 909 may mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area may store an operating system, application programs or instructions (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like. Further, the memory 909 may include a volatile memory or a nonvolatile memory, or the memory 909 may include both volatile and nonvolatile memories. The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable EPROM (EEPROM), or a flash Memory. The volatile memory may be random access memory (Random Access Memory, RAM), static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (ddr SDRAM), enhanced SDRAM (Enhanced SDRAM), synchronous DRAM (SLDRAM), and Direct RAM (DRRAM). Memory 909 in embodiments of the present application includes, but is not limited to, these and any other suitable types of memory.

Processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor that primarily processes operations involving an operating system, user interface, application programs, and the like, and a modem processor that primarily processes wireless communication signals, such as a baseband processor. It will be appreciated that the modem processor described above may not be integrated into the processor 910.

The embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or the instruction is executed by a processor, the processes of the method embodiment of gesture recognition are implemented, and the same technical effects can be achieved, so that repetition is avoided, and no further description is given here.

Wherein the processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium such as a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like.

The embodiment of the application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running a program or an instruction, implementing each process of the gesture recognition method embodiment, and achieving the same technical effect, so as to avoid repetition, and no further description is provided here.

It should be understood that the chips referred to in the embodiments of the present application may also be referred to as system-on-chip chips, chip systems, or system-on-chip chips, etc.

Embodiments of the present application provide a computer program product stored in a storage medium, where the program product is executed by at least one processor to implement the respective processes of the method embodiments of gesture recognition described above, and achieve the same technical effects, and are not described herein in detail to avoid repetition.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solutions of the present application may be embodied essentially or in a part contributing to the prior art in the form of a computer software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the methods described in the embodiments of the present application.

The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those of ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are also within the protection of the present application.

Claims

1. A method of gesture recognition applied to a head-mounted display device, the method comprising:

acquiring an image of a user's hand through the head-mounted display device;

acquiring pose data of a first hand of a user through first equipment, wherein the first hand is a part of the hand of the user which is covered under the view angle of the head-mounted display equipment;

and determining the gesture of the user according to the image and the pose data.

2. The method of claim 1, wherein the obtaining, by the first device, pose data of the first hand of the user comprises:

when the first device comprises a wearable device, determining pose data of the first hand according to the motion data of the user hand, wherein the motion data of the user hand is detected by a motion detection unit in the wearable device;

detecting a millimeter wave signal by a millimeter wave detection unit in a hand-operated device in a case where the first device includes the hand-operated device;

determining pose data of a first hand corresponding to the millimeter wave signal according to the millimeter wave signal;

in the case that the first device comprises a spatial positioning device, acquiring an image of a user's hand or measuring distances of a plurality of target positions of the user's hand by an image acquisition device or a distance measurement device in the spatial positioning, and determining pose data of the first hand according to the image or the distances.

3. The method of claim 1, wherein the first device comprises a wearable device worn on a hand of the user, the image comprising an image of the hand of the user, an image of the wearable device, and an image displayed by the wearable device, the determining a gesture of the user from the image and the pose data comprising:

identifying characteristic points from the image, wherein the characteristic points comprise joint points of the hand of the user, characteristic points of the wearing equipment and characteristic points of the image displayed by the wearing equipment;

and determining the gesture of the user according to the position of the characteristic point and the pose data of the first hand.

4. The method of claim 1, wherein the first device comprises a hand manipulation device;

the hand control equipment is provided with a touch detection unit;

the acquiring, by the head-mounted display device, an image of a hand of a user includes:

and under the condition that the hand of the user does not contact the touch detection area, acquiring an image of the hand of the user through the head-mounted display device.

5. The method of claim 1, wherein prior to obtaining, by the first device, pose data of the first hand of the user, the method further comprises:

Acquiring position information and parameter information of a plurality of space positioning devices around the head-mounted display device; the position information includes a distance and an angle of a spatial positioning device relative to the head mounted display device;

and determining the first equipment from a plurality of space positioning equipment according to the position information and the parameter information.

6. An apparatus for gesture recognition, applied to a head-mounted display device, characterized in that the apparatus comprises:

the first acquisition module is used for acquiring an image of the hand of the user through the head-mounted display device;

the second acquisition module acquires pose data of a first hand of a user through first equipment, wherein the first hand is a part of the hand of the user which is covered under the view angle of the head-mounted display equipment;

and the first determining module is used for determining gestures of a user according to the images and the pose data.

7. The apparatus of claim 6, wherein the second acquisition module comprises:

a first determining submodule, configured to determine pose data of a first hand according to motion data of the user hand detected by a motion detection unit in a wearable device, where the first device includes the wearable device;

A second determination sub-module configured to detect a millimeter wave signal by a millimeter wave detection unit in a hand-manipulation device in a case where the first device includes the hand-manipulation device;

and the third determining submodule is used for acquiring images of the hands of the user or measuring the distances of a plurality of target positions of the hands of the user through the image acquisition equipment or the distance measuring equipment in the space positioning under the condition that the first equipment comprises the space positioning equipment, and determining the pose data of the first hand according to the images or the distances.

8. The apparatus of claim 6, wherein the first device comprises a wearable device worn on the user's hand, the image comprising an image of the user's hand, an image of the wearable device, and an image displayed by the wearable device, the first determination module comprising:

the identification sub-module is used for identifying characteristic points from the image, wherein the characteristic points comprise joint points of the hand of the user, characteristic points of the wearing equipment and characteristic points of the image displayed by the wearing equipment;

And the fourth determining submodule is used for determining gestures of the user according to the positions of the characteristic points and the pose data of the first hand.

9. The apparatus of claim 6, wherein the first device comprises a hand manipulation device;

the hand control equipment is provided with a touch detection unit;

the first acquiring module is specifically configured to acquire, through the head-mounted display device, an image of a hand of the user when the hand of the user does not contact the touch detection area.

10. The apparatus of claim 6, wherein prior to obtaining, by the first device, pose data of the first hand of the user, the apparatus further comprises:

a third acquisition module, configured to acquire position information and parameter information of a plurality of spatial positioning devices around the head-mounted display device; the position information includes a distance and an angle of a spatial positioning device relative to the head mounted display device;

and the second determining module is used for determining the first equipment in the plurality of space positioning equipment according to the position information and the parameter information.

11. An electronic device comprising a processor and a memory storing a program or instructions executable on the processor, which when executed by the processor, perform the steps of the method of gesture recognition as claimed in any one of claims 1 to 5.

12. A computer readable storage medium, characterized in that the readable storage medium has stored thereon a program or instructions which, when executed by a processor, implement the steps of the gesture recognition method according to any of claims 1-5.