WO2021035646A1 - 可穿戴设备及其控制方法、识别手势的方法和控制系统 - Google Patents

可穿戴设备及其控制方法、识别手势的方法和控制系统 Download PDF

Info

Publication number
WO2021035646A1
WO2021035646A1 PCT/CN2019/103440 CN2019103440W WO2021035646A1 WO 2021035646 A1 WO2021035646 A1 WO 2021035646A1 CN 2019103440 W CN2019103440 W CN 2019103440W WO 2021035646 A1 WO2021035646 A1 WO 2021035646A1
Authority
WO
WIPO (PCT)
Prior art keywords
wearable device
key point
point information
control
instruction
Prior art date
Application number
PCT/CN2019/103440
Other languages
English (en)
French (fr)
Inventor
刘志鹏
李思晋
Original Assignee
深圳市大疆创新科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市大疆创新科技有限公司 filed Critical 深圳市大疆创新科技有限公司
Priority to PCT/CN2019/103440 priority Critical patent/WO2021035646A1/zh
Priority to CN201980033213.5A priority patent/CN112154402A/zh
Publication of WO2021035646A1 publication Critical patent/WO2021035646A1/zh
Priority to US17/472,761 priority patent/US11782514B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304—Detection arrangements using opto-electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/251—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language

Definitions

  • This specification relates to the field of human-computer interaction technology, and in particular to a wearable device and its control method, a method for recognizing gestures, a movable platform control system and a storage medium.
  • the traditional interaction mode is realized by the user by touching the touchpad on the wearable device.
  • the user can only place a finger on the touchpad and slide left and right or up and down to select a menu, which is equivalent to only making a selection in a one-dimensional space, and the menu can only be swiped once by sliding the finger.
  • the target button is far away, you need to slide your finger multiple times, which is cumbersome and not fast enough.
  • this manual provides a wearable device and its control method, a method for recognizing gestures, a movable platform control system and a storage medium, aiming to solve the technology of slower and less accurate gesture recognition of wearable devices problem.
  • this specification provides a method for controlling a wearable device, including:
  • a control instruction is generated according to the input instruction to perform task operations according to the control instruction.
  • this specification provides a wearable device, including a memory and a processor
  • the memory is used to store a computer program
  • the processor is configured to execute the computer program and, when executing the computer program, implement the following steps:
  • a control instruction is generated according to the input instruction to perform task operations according to the control instruction.
  • this specification provides a mobile platform control system, including:
  • the movable platform includes an image acquisition device for sending the image taken by the image acquisition device to the wearable device;
  • the aforementioned wearable device is used to display the image sent by the movable platform.
  • this specification provides a method for recognizing gestures, the method including:
  • this specification provides a wearable device, including a memory and a processor
  • the memory is used to store a computer program
  • the processor is configured to execute the computer program and, when executing the computer program, implement the aforementioned method for recognizing gestures.
  • this specification provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and the computer program can be used by a processor to implement the above-mentioned method.
  • the embodiments of this specification provide a wearable device and its control method, a method for recognizing gestures, a movable platform control system and a storage medium, by acquiring the key point information of the gesture action of the target user, and identifying the input instruction according to the key point information , And generate control instructions according to input instructions to perform task operations according to the control instructions; realize that users can quickly use the hand to make gesture actions to control wearable devices, and recognize the input instructions corresponding to the gesture actions through key point information, which can eliminate interference Gesture recognition information is faster and more accurate, which makes it easier for users to control wearable devices more quickly and accurately.
  • FIG. 1 is a schematic flowchart of a control method provided by an embodiment of this specification
  • Figure 2 is a schematic diagram of the key points of gesture actions
  • FIG. 3 is a schematic diagram of an application scenario of an embodiment of a method for controlling a head-mounted display device
  • Figure 4 is a schematic diagram of different types of gestures
  • FIG. 5 is a schematic diagram of an application scenario of another embodiment of a method for controlling a head-mounted display device
  • FIG. 6 is a schematic diagram of an application scenario of another embodiment of a method for controlling a head-mounted display device
  • FIG. 7 is a schematic diagram of a sub-process of an embodiment of obtaining key point information in FIG. 1;
  • FIG. 8 is a schematic diagram of an application scenario of still another embodiment of a method for controlling a head-mounted display device
  • Fig. 9 is a schematic block diagram of a wearable device according to an embodiment of the present specification.
  • FIG. 10 is a schematic flowchart of a method for recognizing gestures according to an embodiment of the present specification.
  • Fig. 11 is a schematic block diagram of a wearable device according to another embodiment of the present specification.
  • FIG. 1 is a schematic flowchart of a method for controlling a wearable device according to an embodiment of the present specification.
  • the control method of the wearable device can be applied to the wearable device to implement corresponding control functions and other processes according to the user's gesture.
  • the wearable device may be, for example, a helmet, a watch, glasses, a jacket, a belt, a protective belt, etc. having an image capturing device.
  • the wearable device may be a head-mounted display device.
  • the head-mounted display device may be a virtual reality (VR, virtual reality) display device or a first person view (FPV, first person view) display device.
  • VR virtual reality
  • FV first person view
  • the head-mounted display device may be, for example, a glasses-type display device or a helmet-type display device.
  • the control method of the wearable device includes steps S110 to S130.
  • S110 Acquire key point information of the gesture action of the target user.
  • the triangle in FIG. 2 is used to identify key points on the target user's hand, and the key points include, for example, finger joints, finger tips, and wrist joints.
  • the key point information includes location information of the key point.
  • control method is applied to a head-mounted display device. According to this specification, it can be understood that the control method is applied to various wearable devices.
  • step S110 acquiring the key point information of the target user's gesture action includes: acquiring a user image taken by an image acquisition device equipped with a head-mounted display device; according to the user image taken by the image acquisition device equipped with the head-mounted display device The image determines the key point information of the gesture action.
  • FIG. 3 is a schematic diagram of an application scenario of an embodiment of a method for controlling a head-mounted display device.
  • the target user wears a head-mounted display device, which has its own image acquisition device or is connected to the image acquisition device through a communication interface.
  • the image acquisition device is located on the lower side, upper side, left side, or right side of the head-mounted display device.
  • the image acquisition device is, for example, an independent camera or an electronic device such as a mobile phone or a camera.
  • the target user extends his hand and is located within the shooting field of view of the image acquisition device mounted on the head-mounted display device. Therefore, the head-mounted display device can acquire the hand image of the target user through the image acquisition device. Then, the head-mounted display device can determine the key points in the hand image according to the hand image of the target user, and obtain the key point information of the gesture action as shown in FIG. 2.
  • the user image taken by the image acquisition device includes a background area and a hand area.
  • the head-mounted display device may first identify the hand area in the user image, and then obtain key point information in the hand area.
  • the head-mounted display device After the head-mounted display device acquires the image taken by the image acquisition device, it performs hand detection in the entire image, and outputs the specific position of the hand.
  • a target frame may be used to identify the position of the hand.
  • the target frame includes the coordinates of the upper left corner and the lower right corner of the target frame on the captured image; or the target frame includes the coordinates of the center of the target frame on the captured image and the width and height of the target frame.
  • the image captured by the image acquisition device of the head-mounted display device may include one hand area, or may include two or more hand areas.
  • the head-mounted display device crops a picture of the hand area from the image taken by the image acquisition device according to the specific position of the hand. Then input the picture of the hand area into the hand key point detection neural network to get the hand key points in the picture.
  • the hand key point detection neural network can be obtained by training the deep neural network to convergence according to the hand key point sample set.
  • the hand key point sample set includes a plurality of hand pictures with key points, and these hand pictures correspond to a variety of different gestures. Therefore, the trained hand key point detection neural network can more accurately detect the hand key points from the picture of the hand area.
  • the image acquisition device mounted on the head-mounted display device includes at least one of a color camera, a depth sensor, and an RGB-D camera.
  • the images captured by the depth sensor or RGB-D camera have depth characteristics, and the key points of the hand can be detected more accurately based on the images captured by the depth sensor or RGB-D camera.
  • the head-mounted display device is equipped with at least two image acquisition devices.
  • the head-mounted display device determines the key point information of the gesture action according to the user images taken by the at least two image acquisition devices mounted thereon.
  • the at least two image acquisition devices are located at different positions on the head-mounted display device or although they are located at the same position, the shooting angles are different. In this way, the user image of the target user can be obtained in different viewing angles, so as to determine the hand image of the target user in different viewing angles. You can get richer information about the key points of gesture actions.
  • multiple image acquisition devices can respectively acquire key point information on the palm and the back of the hand, so as to prevent the gesture recognition accuracy rate from being reduced due to the occlusion of the key points of the hand.
  • the type of gesture action can be determined according to the key point information of the gesture action.
  • Figure 4 shows a schematic diagram of different types of gestures.
  • the types of gesture actions include one-hand gesture actions or two-hand gesture actions.
  • the key point information is sent to a gesture action classification model; the key point information is classified based on the gesture action classification model to obtain an input instruction.
  • the gesture action classification model can be obtained by training the deep neural network to convergence according to the gesture sample set.
  • the gesture sample set includes several key point information and gesture annotations corresponding to the key point information. These key points of information correspond to a variety of different gestures. Therefore, the trained gesture action classification model can more accurately identify the gesture actions of the target user based on the key point information.
  • the head-mounted display device pre-stores a mapping table of gesture actions and input instructions.
  • the mapping table includes the correspondence between several gesture actions and several input instructions. Therefore, the head-mounted display device can determine the corresponding input instruction according to the recognized gesture action of the target user.
  • the interface displayed by the head-mounted display device includes the correspondence between three gesture actions and three input instructions in the upper right corner.
  • the user needs to input an instruction, he can make a corresponding gesture with his hand in front of the image acquisition device according to the displayed corresponding relationship.
  • the head-mounted display device may recognize the gesture action of the target user, and determine that the input instruction is the mode switching instruction according to the gesture action.
  • the key point information is sent to an instruction classification model; the key point information is classified based on the instruction classification model to obtain an input instruction.
  • the instruction classification model can be obtained by training the deep neural network to convergence according to the gesture sample set.
  • the gesture sample set includes several key point information and instruction labels corresponding to each of the key point information. These key points of information correspond to a variety of different gestures. Therefore, the trained gesture action classification model can more accurately identify the input instructions of the target user based on the key point information.
  • the head-mounted display device may not recognize the gesture action, but directly recognize the input instruction according to the key point information.
  • the mapping table of gesture actions and input instructions may not be stored.
  • the body interaction mode is turned on or off according to the mode switching command.
  • the body interaction mode is used to perform task operations according to the control instructions generated by the recognized input instructions.
  • the target user when the target user wants to turn on the body interaction mode to control the head-mounted display device to interact according to the gesture actions made by the target user, the target user may make a gesture action corresponding to the mode switching instruction. Therefore, the head-mounted display device can turn on the body interaction mode according to the determined mode switching instruction.
  • the head-mounted display device When the body interaction mode is turned on, the head-mounted display device obtains key point information of the gesture action of the target user, and recognizes the input instruction according to the key point information. If it is recognized that the input command is a mode switching command again, the body interaction mode is closed. After turning off the body interaction mode, if the recognized input command is not a mode switching command, it will not respond to the input command; if the recognized input command is a mode switching command, then the body interaction mode will be turned on.
  • the head-mounted display device When the body interaction mode is turned on, the head-mounted display device responds to the recognized input instruction.
  • different input instructions correspond to different control instructions.
  • the head-mounted display device pre-stores a mapping table of input instructions and control instructions.
  • the mapping table includes the correspondence between a number of input instructions and a number of control instructions.
  • the user can associate the input instruction with a function specified by the user through a setting operation, such as saving a screenshot function of the screen displayed by the head-mounted display device.
  • the head-mounted display device can adjust the corresponding relationship between the input instruction and the control instruction according to the user's setting operation, and the control instruction corresponds to the function specified by the user.
  • the head-mounted display device can determine the corresponding control instruction according to the recognized input instruction. Then, a control instruction is generated according to the input instruction to perform task operations according to the control instruction.
  • the head-mounted display device displays an interactive operation interface.
  • the executing the task operation according to the control instruction includes: switching the interactive operation interface displayed by the head-mounted display device according to the control instruction.
  • the input instruction is an instruction to switch to the next interactive operation interface
  • a command for controlling the head-mounted display device to switch to the next interactive operation interface is generated.
  • the control instruction is used to switch the head-mounted display device to the next interactive operation interface.
  • a control instruction for controlling the head-mounted display device to switch to the next interactive operation interface is generated according to the currently displayed interactive operation interface and the input instruction, so that the head-mounted display device is switched Go to the next interactive operation interface.
  • a control instruction for controlling the head-mounted display device to switch to the previous interactive operation interface is generated according to the currently displayed interactive operation interface and the input instruction, so that the head-mounted display device returns to the previous interface.
  • An interactive operation interface For example, if the input instruction is a return instruction, a control instruction for controlling the head-mounted display device to switch to the previous interactive operation interface is generated according to the currently displayed interactive operation interface and the input instruction, so that the head-mounted display device returns to the previous interface.
  • the generating the control instruction according to the input instruction includes: determining operation control information according to the interactive operation interface currently displayed by the head-mounted display device and the input instruction; generating corresponding control information according to the operation control information instruction.
  • the head-mounted display device displays a certain interactive interface
  • it is allowed to save the screen displayed by the head-mounted display device according to the user's gesture action.
  • the head-mounted display device displays the interactive interface
  • the input instruction identified according to the key point information is a confirmation instruction
  • it can be determined according to the input instruction that the operation control information corresponds to the screenshot function.
  • a control instruction corresponding to the screenshot function is generated, so that the head-mounted display device saves the currently displayed picture.
  • the interactive operation interface displays one virtual button or multiple virtual buttons, and the virtual buttons correspond to different operation control information.
  • an interactive operation interface includes virtual buttons corresponding to the screenshot function.
  • the input instruction identified according to the key point information is a confirmation instruction
  • a control instruction corresponding to the screenshot function is generated, so that the head-mounted display device saves the currently displayed picture.
  • the determining operation control information according to the interactive operation interface and the input instruction includes: determining a function button in the interactive operation interface; and determining operation control information according to the function button and the input instruction.
  • the head-mounted display device is communicatively connected to a movable platform, such as an unmanned aerial vehicle.
  • the interactive operation interface includes multiple function buttons, such as a return button corresponding to controlling the return of the unmanned aerial vehicle, a hover button corresponding to controlling the hover of the unmanned aerial vehicle, and a screenshot corresponding to the screenshot function. Buttons and menu buttons.
  • the function button selected by the user in the interactive operation interface is determined, and then the operation control information is determined according to the function button selected by the user and the input instruction.
  • the head-mounted display device determines the position of the preset part of the target user's hand on the interactive operation interface according to the key point information of the target user's gesture action; determines the position of the preset part of the target user's hand on the interactive operation interface; Description of the function buttons.
  • the head mounted display device determines the position of the index finger of the target user on the interactive operation interface according to the key point information of the gesture action. Specifically, the position of the cursor corresponding to the fingertip of the index finger on the interactive operation interface is determined according to the position of the fingertip of the index finger of the user in the image of the user captured by the image acquisition device equipped with the display device.
  • the user can control the cursor in the interactive operation interface by moving the index finger of the left or right hand.
  • the cursor moves to a certain function button in the interactive operation interface, it is determined that the function button is a function button that can respond to the input instruction.
  • the head-mounted display device can respond to input commands to trigger the functions of the function buttons, such as controlling the UAV to hover or return to home.
  • a function button determined on an interactive operation interface is a function button corresponding to a return home instruction
  • the input instruction identified according to the key point information is a confirmation instruction
  • the operation control information corresponding to the return home can be determined according to the input instruction.
  • a control instruction is generated, so that the head-mounted display device sends a return instruction to the unmanned aerial vehicle according to the control instruction, thereby controlling the unmanned aerial vehicle to return home.
  • the performing task operations according to the control instruction includes: adjusting a menu in an interactive operation interface currently displayed on the head-mounted display device.
  • the interactive operation interface includes a menu button.
  • the function button determined by the interactive operation interface is the menu button
  • the operation control information corresponding to the expanded menu can be determined according to the input instruction.
  • a control instruction is generated according to the operation control information, so that the head-mounted display device displays option buttons corresponding to the menu on the interactive operation display interface, such as option one and option two.
  • the head-mounted display device determines the position of the preset part of the target user's hand on the interactive operation interface according to the key point information of the target user's gesture action; determines the corresponding option 2 according to the position on the interactive operation interface Function button. If the input instruction identified later based on the key point information is a confirmation instruction, the operation control information corresponding to the second option can be determined according to the input instruction. A control instruction is generated according to the operation control information, so that the head-mounted display device implements the operation corresponding to option two.
  • the head-mounted display device is communicatively connected to a movable platform.
  • the movable platform is an unmanned aerial vehicle, an unmanned vehicle or an unmanned boat.
  • the unmanned aerial vehicle can be, for example, a rotary wing drone, such as a quadrotor drone, a hexarotor drone, an eight rotor drone, or a fixed wing drone.
  • the execution of the task operation according to the control instruction includes: starting or ending storage of the data acquired by the head-mounted display device from the movable platform according to the control instruction.
  • the head-mounted display device acquires data from the movable platform through a communication link with the movable platform, such as an image taken by an image acquisition device mounted on the movable platform. After the head-mounted display device generates a certain control instruction, it starts to store or ends to store the data obtained from the movable platform according to the control instruction.
  • the execution of the task operation according to the control instruction includes: sending a platform control instruction to a movable platform according to the control instruction to control the movable platform.
  • the head-mounted display device sends a platform control instruction for controlling the return home of the movable platform to the movable platform according to the control instruction. If the control instruction corresponding to the accelerated movement is generated according to the input instruction, the head mounted display device sends a platform control instruction for controlling the accelerated movement of the movable platform to the movable platform according to the control instruction.
  • control method is applied to a watch, and the watch includes an image acquisition device, for example, a camera.
  • the watch obtains the key point information of the gesture action of the target user through the camera, and then recognizes the input instruction according to the key point information, and generates a control instruction according to the input instruction to perform task operations according to the control instruction.
  • the watch can also display an interactive operation interface.
  • the watch may also switch the displayed interactive operation interface according to the control instruction, determine operation control information according to the currently displayed interactive operation interface and the input instruction, and generate corresponding control instructions based on the operation control information.
  • the watch can also be communicatively connected to a movable platform, and the watch can respond to input commands to trigger the functions of the function buttons, such as controlling the unmanned aerial vehicle to hover or return to home.
  • the head-mounted display device is connected to a movable platform, and the movable platform is equipped with an image acquisition device.
  • the movable platform is connected to the movable platform through a wired method or connected to the movable platform through a wireless communication method.
  • the movable platform has its own image acquisition device or is connected to the image acquisition device through a communication interface.
  • the image acquisition device includes at least one of a color camera, a depth sensor, and an RGB-D camera.
  • the image acquisition device is, for example, an independent camera or an electronic device such as a mobile phone or a camera.
  • step S110 acquiring key point information of the gesture action of the target user includes: determining the location based on the user image taken by the image acquisition device mounted on the movable platform and the user image taken by the image acquisition device mounted on the head-mounted display device. Describe the key point information of the gesture action.
  • the head-mounted display device can determine the key point information of the gesture action according to the user image taken by the head-mounted display device and the user image taken by the movable platform.
  • user images of the target user under different viewing angles can be obtained, so as to determine the hand images of the target user under different viewing angles.
  • More abundant key point information of gesture actions can be obtained, and the accuracy of gesture recognition can be prevented from being occluded by the key points of the hand.
  • the key point information on the palm and back of the hand of the target user can be determined based on the user image taken by the head-mounted display device and the user image taken by the movable platform.
  • the head-mounted display device is connected to a handheld control device, and the handheld control device is equipped with an image acquisition device.
  • the handheld control device is connected to the handheld control device in a wired manner or connected to the handheld control device in a wireless communication manner.
  • the head-mounted display device may also perform corresponding operations or functions according to the control instructions sent by the handheld control device, such as switching the interactive operation interface, determining the function buttons in the interactive operation interface, and turning on or off the body interaction mode.
  • the handheld control device has its own image acquisition device or is connected to the image acquisition device through a communication interface.
  • the image acquisition device includes at least one of a color camera, a depth sensor, and an RGB-D camera.
  • the image acquisition device is, for example, an independent camera or an electronic device such as a mobile phone or a camera.
  • step S110 acquiring key point information of the gesture action of the target user includes: determining the location based on the user image taken by the image acquisition device equipped with the handheld control device and the user image taken by the image acquisition device equipped with the head-mounted display device. Describe the key point information of the gesture action.
  • the head-mounted display device can determine the key point information of the gesture action based on the user image taken by the head-mounted display device and the user image taken by the handheld control device.
  • user images of the target user under different viewing angles can be obtained, so as to determine the hand images of the target user under different viewing angles.
  • More abundant key point information of gesture actions can be obtained, and the accuracy of gesture recognition can be prevented from being occluded by the key points of the hand.
  • the key point information on the palm and back of the hand of the target user can be determined based on the user image taken by the head-mounted display device and the user image taken by the handheld control device.
  • the wearable device such as a watch
  • the wearable device may also be communicatively connected to a movable platform and/or a handheld control device.
  • Wearable devices such as watches can also determine key point information of gesture actions based on user images taken by the wearable device and user images taken by the handheld control device and/or the movable platform.
  • obtaining the key point information of the gesture action of the target user in step S110 includes step S111 to step S114.
  • S111 Acquire an image of a user captured by an image acquisition device mounted on the head-mounted display device.
  • S112 Determine whether the key point information of the gesture action of the target user is occluded according to the user image taken by the image acquisition device mounted on the head-mounted display device.
  • the key points in the hand image are determined according to the user image taken by the image acquisition device equipped with the head-mounted display device. If the number of key points is less than a preset threshold, such as 22, it is determined that the key point information of the gesture action is blocked ; If the number of key points is not less than the preset threshold, it is determined that the key point information of the gesture action is not blocked.
  • a preset threshold such as 22
  • the key point information of the gesture action is determined according to the user image taken by the image acquisition device mounted on the head-mounted display device.
  • S113 If the key point information is blocked, acquire a user image taken by an image acquisition device mounted on the handheld control device and/or acquire a user image taken by an image acquisition device mounted on the movable platform.
  • the head-mounted display device is connected to the movable platform and/or the handheld control device.
  • the movable platform and/or the handheld control device are each equipped with an image acquisition device.
  • the image determines the key point information of the gesture action.
  • the head-mounted display device can determine the key point information of the gesture action based on the user image taken by the head-mounted display device and the user image taken by the movable platform and/or the handheld control device. In this way, user images of the target user under different viewing angles can be obtained, so as to determine the hand images of the target user under different viewing angles. More abundant key point information of gesture actions can be obtained, and the accuracy of gesture recognition can be prevented from being occluded by the key points of the hand. For example, you can determine the key point information on the palm and back of the target user's hand.
  • obtaining the key point information of the target user's gesture action in step S110 includes: obtaining a user image taken by an image obtaining device equipped on a movable platform; and according to the user taken by the image obtaining device equipped on the movable platform The image determines whether the key point information of the target user's gesture is occluded. If it is determined that the key point information of the gesture action is not blocked, the key point information of the gesture action is determined according to the user image taken by the image acquisition device mounted on the movable platform. If the key point information is occluded, the user image taken by the image acquisition device mounted on a wearable device, such as a head-mounted display device, a smart watch, etc., is acquired. Then, the key point information of the gesture action is determined according to the user image taken by the image acquisition device mounted on the wearable device and the user image taken by the image acquisition device mounted on the movable platform.
  • the movable platform obtains the recognition result of the gesture according to the key point information of the gesture action, and performs corresponding operations according to the recognition result, such as detecting the shooting target in the image taken by the image acquisition device mounted on the movable platform , Take pictures of the target, etc.
  • the movable platform or the wearable device can determine the key point information of the gesture action according to the user image taken by the wearable device and the user image taken by the movable platform. In this way, user images of the target user under different viewing angles can be obtained, so as to determine the hand images of the target user under different viewing angles. More abundant key point information of gesture actions can be obtained, and the accuracy of gesture recognition can be prevented from being occluded by the key points of the hand. For example, you can determine the key point information on the palm and back of the target user's hand.
  • the target user holds a handheld control device and wears a head-mounted display device; the movable platform can perform corresponding actions or functions according to instructions sent by the handheld control device and/or head-mounted display device .
  • the user can slightly leave the hand-held control device and make a corresponding gesture action, and then input the corresponding instruction of the gesture action to the head-mounted display device, so that the head-mounted display device generates a control instruction according to the input instruction, and executes it according to the control instruction Task operation. Therefore, the user does not need to raise his hand to the touchpad of the head-mounted display device worn on his head for control, which saves more effort.
  • the user can directly select and trigger the menu on the two-dimensional by hand.
  • the cursor on the interface can be controlled to move left and right or up and down, and the cursor can also be controlled to sit up and right. Move in different directions, the operation is faster and the experience is more friendly.
  • the control method of the head-mounted display device obtains the key point information of the gesture action of the target user, recognizes the input instruction according to the key point information, and generates the control instruction according to the input instruction, so as to perform the task operation according to the control instruction ; Realize that the user can quickly use the hand to make gesture actions to control the head-mounted display device, and recognize the input instructions corresponding to the gesture action through the key point information, which can eliminate the information that will interfere with the gesture recognition, and the recognition speed is faster and more accurate, which is convenient The user can control the head-mounted display device more quickly and accurately.
  • the key point information of the gesture action can be determined through the user image taken by the head-mounted display device and the user image taken by the movable platform and/or the handheld control device, so as to obtain more abundant key point information of the gesture action, and can prevent The occlusion of the key points of the hand leads to a decrease in the accuracy of gesture recognition.
  • the user can "press" the virtual key in the interactive operation interface through a specific gesture to trigger the corresponding key function.
  • a specific gesture such as "scissors hand”
  • he can "press” the virtual button at the cursor position to control the head-mounted display device or the movable platform connected through the head-mounted display device controller and or Handheld control device.
  • FIG. 9 is a schematic block diagram of a wearable device 600 according to an embodiment of the present specification.
  • the wearable device 600 may be a virtual reality (VR, virtual reality) display device or a first person view (FPV, first person view) display device, etc.
  • the wearable device 600 may be, for example, a glasses-type display device or a helmet-type display device.
  • the wearable device 600 includes a processor 601 and a memory 602, and the processor 601 and the memory 602 are connected by a bus 603.
  • the bus 603 is, for example, an I2C (Inter-integrated Circuit) bus.
  • the processor 601 may be a micro-controller unit (MCU), a central processing unit (Central Processing Unit, CPU), a digital signal processor (Digital Signal Processor, DSP), or the like.
  • MCU micro-controller unit
  • CPU Central Processing Unit
  • DSP Digital Signal Processor
  • the memory 602 may be a Flash chip, a read-only memory (ROM, Read-Only Memory) disk, an optical disk, a U disk, or a mobile hard disk.
  • the wearable device 600 has its own image acquisition device 604 or is connected to the image acquisition device 604 through a communication interface.
  • the image acquisition device 604 is located on the lower side, upper side, left side, or right side of the wearable device 600.
  • the processor 601 is configured to run a computer program stored in the memory 602, and implement the aforementioned control method when the computer program is executed.
  • FIG. 5 is a schematic diagram of a movable platform control system provided by an embodiment of this specification.
  • the movable platform control system includes a movable platform and a wearable device.
  • the movable platform includes an image acquisition device, which is used to send the image taken by the image acquisition device to a wearable device; and the wearable device is used to display the image sent by the movable platform.
  • the movable platform control system further includes: a handheld control device for sending remote control instructions to the movable platform.
  • the movable platform is also used to move according to the remote control instruction.
  • FIG. 10 is a schematic flowchart of a method for recognizing gestures according to an embodiment of this specification.
  • the method of recognizing gestures can be applied to wearable devices and mobile platforms equipped with image acquisition devices.
  • Wearable devices can be, for example, head-mounted display devices, smart watches, tops, belts, protective belts, etc.; head-mounted display devices can be virtual reality (VR, virtual reality) display devices or first-person view (FPV, first person view) displays equipment.
  • the head-mounted display device may be, for example, a glasses-type display device or a helmet-type display device.
  • the movable platform may be, for example, an unmanned aerial vehicle equipped with an image acquisition device, a handheld PTZ, a mobile robot, a vehicle, and the like.
  • the method for recognizing a gesture includes step S210 to step S250.
  • the first image is obtained by an image acquisition device mounted on a wearable device.
  • an image of a user captured by an image acquisition device mounted on a wearable device is acquired.
  • the wearable device is a helmet, a watch, glasses, a jacket, a belt, or a protective belt with an image acquisition device.
  • S220 Obtain key point information of the gesture according to the first image.
  • the hand detection is performed in the entire image range, and the specific position of the hand is output.
  • a picture of the hand area is cropped from the image taken by the camera.
  • input the picture of the hand area into the hand key point detection neural network to obtain the key point information of the gesture, such as the number and position of the hand key point in the picture.
  • a preset threshold such as 22
  • the number of key points is not less than the preset threshold, it is determined that the key point information of the gesture action is not blocked, and the recognition result of the gesture can be obtained according to the key point information.
  • the second image is obtained by another image acquisition device mounted on the wearable device.
  • the head-mounted display device is equipped with at least two cameras in different positions.
  • the head-mounted display device determines the key point information of the gesture action according to user images taken by at least two cameras mounted in different positions. In this way, the user image of the target user can be obtained in different viewing angles, so as to determine the hand image of the target user in different viewing angles. You can get richer information about the key points of gesture actions.
  • the key point information on the palm of the hand and the back of the hand can be obtained according to the images of multiple cameras, so as to prevent the hand key points from being occluded and the accuracy of gesture recognition is reduced.
  • the position information of the key points in the first image and the position information of the key points in the second image are stored together as the key point information of the gesture.
  • the type of the gesture action can be determined.
  • the key point information is sent to a gesture action classification model; the key point information is classified based on the gesture action classification model to obtain a recognition result of the gesture.
  • the first image is obtained by an image acquisition device mounted on a movable platform.
  • the movable platform is an unmanned aerial vehicle with an image acquisition device, a handheld pan/tilt, a mobile robot, and a vehicle.
  • the second image is obtained by an image acquisition device mounted on a wearable device.
  • the method for recognizing gestures obtained by the embodiments of this specification obtains a second image of the gesture when the type of gesture cannot be determined according to the key point information of the first image, and updates the key point information of the gesture according to the second image, so as to update the key point information according to the updated
  • the key point information of to obtain the recognition result of the gesture More abundant key point information of gesture actions can be obtained, and the accuracy of gesture recognition can be prevented from being occluded by the key points of the hand.
  • FIG. 11 is a schematic block diagram of a wearable device 70 according to an embodiment of this specification.
  • the wearable device 70 includes a memory 71 and a processor 72; the memory 71 is used to store a computer program; the processor 72 is used to execute the computer program and when the computer program is executed, Implement the aforementioned method of recognizing gestures.
  • the wearable device includes at least one of the following: a head-mounted display device, a smart watch, a coat, a waist belt, and a protective belt.
  • the embodiments of this specification also provide a computer-readable storage medium, the computer-readable storage medium stores a computer program, the computer program includes program instructions, and the processor executes the program instructions to implement the foregoing implementation
  • the steps of the control method and/or the method of recognizing gestures are provided in the example.
  • the computer-readable storage medium may be the internal storage unit of the wearable device described in any of the foregoing embodiments, such as the hard disk or memory of the wearable device.
  • the computer-readable storage medium may also be an external storage device of the wearable device, such as a plug-in hard disk equipped on the wearable device, a smart memory card (Smart Media Card, SMC), and Secure Digital (Secure Digital). , SD) card, flash card (Flash Card), etc.
  • the wearable device, the method for recognizing gestures, the movable platform control system, and the computer-readable storage medium provided by the above-mentioned embodiments of this specification acquire the key point information of the gesture action of the target user, and recognize the input instruction according to the key point information, and Generate control instructions according to input instructions to perform task operations according to the control instructions; realize that users can quickly use the hand to make gesture actions to control wearable devices, and recognize the input instructions corresponding to the gesture actions through key point information, which can eliminate interference with gesture recognition
  • the recognition speed is faster and more accurate, which makes it easier for users to control wearable devices more quickly and accurately.
  • the key point information of the gesture action can be determined through the user image taken by the wearable device and the user image taken by the movable platform and/or the handheld control device, so that more abundant key point information of the gesture action can be obtained, and the hand can be prevented. Some key points are occluded, which reduces the accuracy of gesture recognition.
  • the user can "press" the virtual key in the interactive operation interface through a specific gesture to trigger the corresponding key function.
  • a specific gesture such as "scissors hand”
  • he can "press” the virtual button at the cursor position to control the wearable device or a movable platform and or handheld control connected through the wearable device controller Device.

Abstract

一种可穿戴设备及其控制方法、识别手势的方法、可移动平台控制系统和存储介质，包括：获取目标用户的手势动作的关键点信息(S110)；根据所述关键点信息识别输入指令(S120)；根据所述输入指令产生控制指令，以根据所述控制指令执行任务操作(S130)。

Description

可穿戴设备及其控制方法、识别手势的方法和控制系统 技术领域
本说明书涉及人机交互技术领域，尤其涉及一种可穿戴设备及其控制方法、识别手势的方法、可移动平台控制系统和存储介质。
背景技术
在可穿戴设备上，传统的交互模式是用户通过触摸可穿戴式设备上的触控板实现的。用户只能将手指放在触控板上进行左右或上下滑动来进行菜单的选择，相当于只能在一维空间中进行选择，并且滑动一次手指只能将菜单滑动一次。此时如果目标按钮比较远，则需要多次滑动手指，过程较为繁琐，不够快捷。虽然一些研究提出根据可穿戴设备的图像获取装置拍摄的用户手势识别控制指令，但是这种识别方式速度较慢、准确性较低。
发明内容
基于此，本说明书提供了一种可穿戴设备及其控制方法、识别手势的方法、可移动平台控制系统和存储介质，旨在解决可穿戴设备的手势识别速度较慢、准确性较低等技术问题。
第一方面，本说明书提供了一种可穿戴设备的控制方法，包括：
获取目标用户的手势动作的关键点信息；
根据所述关键点信息识别输入指令；
根据所述输入指令产生控制指令，以根据所述控制指令执行任务操作。
第二方面，本说明书提供了一种可穿戴设备，包括存储器和处理器；
所述存储器用于存储计算机程序；
所述处理器，用于执行所述计算机程序并在执行所述计算机程序时，实现如下步骤：
获取目标用户的手势动作的关键点信息；
根据所述关键点信息识别输入指令；
根据所述输入指令产生控制指令，以根据所述控制指令执行任务操作。
第三方面，本说明书提供了一种可移动平台控制系统，包括：
可移动平台，包括图像获取装置，用于将图像获取装置拍摄的图像发送给可穿戴设备；
前述的可穿戴设备，用于显示所述可移动平台发送的图像。
第四方面，本说明书提供了一种识别手势的方法，所述方法包括：
获得所述手势的第一图像；
根据所述第一图像获得所述手势的关键点信息；
当根据所述关键点信息无法判断所述手势类型时，获得所述手势的第二图像；
根据所述第二图像更新所述手势的关键点信息；
根据所述更新后的关键点信息获得所述手势的识别结果。
第五方面，本说明书提供了一种可穿戴设备，包括存储器和处理器；
所述存储器用于存储计算机程序；
所述处理器，用于执行所述计算机程序并在执行所述计算机程序时，实现前述的识别手势的方法。
第六方面，本说明书提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序可被处理器以实现上述的方法。
本说明书实施例提供了一种可穿戴设备及其控制方法、识别手势的方法、可移动平台控制系统和存储介质，通过获取目标用户的手势动作的关键点信息，并根据关键点信息识别输入指令，以及根据输入指令产生控制指令，以根据控制指令执行任务操作；实现用户可以快捷的使用手悬空作出手势动作控制可穿戴设备，且通过关键点信息识别手势动作对应的输入指令，可以排除会干扰手势识别的信息，识别速度更快且更准确，从而方便用户更快捷和更准确的控制可穿戴设备。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本说明书的公开内容。
附图说明
为了更清楚地说明本说明书实施例技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图是本说明书的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本说明书一实施例提供的控制方法的流程示意图；
图2是手势动作的关键点的示意图；
图3是头戴显示设备的控制方法一实施方式的应用场景示意图；
图4是不同种类手势动作的示意图；
图5是头戴显示设备的控制方法另一实施方式的应用场景示意图；
图6是头戴显示设备的控制方法又一实施方式的应用场景示意图；
图7是图1中获取关键点信息一实施方式的子流程示意图；
图8是头戴显示设备的控制方法再一实施方式的应用场景示意图；
图9是本说明书一实施例提供的一种可穿戴设备的示意性框图。
图10是本说明书一实施例提供的一种识别手势的方法的流程示意图。
图11是本说明书另一实施例提供的一种可穿戴设备的示意性框图。
具体实施方式
下面将结合本说明书实施例中的附图，对本说明书实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本说明书一部分实施例，而不是全部的实施例。基于本说明书中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本说明书保护的范围。
附图中所示的流程图仅是示例说明，不是必须包括所有的内容和操作/步骤，也不是必须按所描述的顺序执行。例如，有的操作/步骤还可以分解、组合或部分合并，因此实际执行的顺序有可能根据实际情况改变。
下面结合附图，对本说明书的一些实施方式作详细说明。在不冲突的情况下，下述的实施例及实施例中的特征可以相互组合。
请参阅图1，图1是本说明书一实施例提供的一种可穿戴设备的控制方法的流程示意图。所述可穿戴设备的控制方法可以应用在可穿戴设备中，用于根据用户的手势实现相应的控制功能等过程。
可穿戴设备例如可以为具有图像获取装置的头盔、手表、眼镜、上衣、腰带、护带等。
示例性的，可穿戴设备可以为头戴显示设备。
头戴显示设备可以为虚拟现实(VR，virtual reality)显示设备或第一人称视角(FPV，first person view)显示设备。头戴显示设备例如可以为眼镜式显示设备或者头盔式显示设备等。
如图1所示，可穿戴设备的控制方法包括步骤S110至步骤S130。
S110、获取目标用户的手势动作的关键点信息。
图2中的三角形用于标识目标用户手上的关键点，关键点例如包括手指关节、手指尖、腕关节等。关键点信息包括关键点的位置信息。
本说明书以控制方法应用于头戴显示设备进行说明，根据本说明书可以理解控制方法应用于各种可穿戴设备的实施方式。
在一些实施方式中，步骤S110获取目标用户的手势动作的关键点信息包括：获取头戴显示设备搭载的图像获取装置拍摄的用户图像；根据所述头戴显示设备搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
如图3所示为头戴显示设备的控制方法一实施方式的应用场景示意图。
目标用户佩戴头戴显示设备，头戴显示设备自带图像获取装置或者通过通信接口连接图像获取装置。例如，图像获取装置位于头戴显示设备的下侧、上侧、左侧或右侧。图像获取装置例如为独立的摄像头或者为手机、相机等电子设备。
如图3所示，目标用户将手伸出，位于头戴显示设备搭载的图像获取装置的拍摄视野内。从而头戴显示设备可以通过图像获取装置获取目标用户的手部图像。然后头戴显示设备可以根据目标用户的手部图像确定所述手部图像中的关键点，得到如图2所示的手势动作的关键点信息。
示例性的，图像获取装置拍摄的用户图像包括背景区域和手部区域。头戴显示设备可以先识别所述用户图像中的手部区域，然后获取所述手部区域中的关键点信息。
具体的，头戴显示设备获取图像获取装置拍摄的图像之后，在全图范围内进行手部检测，输出手部的具体位置，例如可以用目标框标识手部位置。目标框包括目标框的左上角和右下角在拍摄的图像上的坐标；或者目标框包括目标 框的中心在拍摄的图像上的坐标以及目标框的宽和高。
具体的，头戴显示设备的图像获取装置拍摄的图像中可以包括一个手部区域，也可以包括两个或两个以上的手部区域。
之后，头戴显示设备根据手部的具体位置从图像获取装置拍摄的图像中裁剪出手部区域的图片。再将手部区域的图片输入到手部关键点检测神经网络得到图片中手部关键点。
手部关键点检测神经网络可以通过根据手部关键点样本集对深度神经网络进行训练至收敛得到。手部关键点样本集包括多个标注有关键点的手部图片，这些手部图片对应于多种不同手势。因此训练好的手部关键点检测神经网络可以更准确的从手部区域的图片中检测到手部关键点。
在一些实施方式中，所述头戴显示设备搭载的图像获取装置包括彩色摄像头、深度传感器、RGB-D摄像头中的至少一种。
深度传感器或RGB-D摄像头拍摄的图像具有深度特征，根据深度传感器或RGB-D摄像头拍摄的图像可以更加精确的进行手部关键点的检测。
在一些实施例中，所述头戴显示设备搭载至少两个图像获取装置。头戴显示设备根据搭载的至少两个图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。所述至少两个图像获取装置位于头戴显示设备上的不同位置或者虽然位于相同位置但拍摄角度不同。从而可以在不同视角下获取目标用户的用户图像，从而确定不同视角下目标用户的手部图像。可以得到更丰富的手势动作的关键点信息。
示例性的，多个图像获取装置可以分别获取手心和手背上的关键点信息，防止手部关键点部分被遮挡导致的手势识别准确率降低。
S120、根据所述关键点信息识别输入指令。
在一些实施方式中，根据手势动作的关键点信息可以确定手势动作的种类。如图4所示为不同种类手势动作的示意图。
如图4所示，手势动作的种类包括一只手的手势动作，或者两只手的手势动作。
在一些实施方式中，将所述关键点信息发送给手势动作分类模型；基于所述手势动作分类模型对所述关键点信息进行分类，以得到输入指令。
示例性的，手势动作分类模型可以通过根据手势样本集对深度神经网络进 行训练至收敛得到。手势样本集包括若干关键点信息和所述关键点信息各自对应的手势标注。这些关键点信息对应于多种不同的手势。因此训练好的手势动作分类模型可以更准确的根据关键点信息识别目标用户的手势动作。
头戴显示设备预先存储了手势动作和输入指令的映射表。该映射表包括若干手势动作和若干输入指令之间的对应关系。从而头戴显示设备可以根据识别出的目标用户的手势动作确定对应的输入指令。
如图3所示，头戴显示设备显示的界面中在右上角包括三种手势动作和三种输入指令的对应关系。用户可以在需要输入指令时，可以根据显示的对应关系，将手在图像获取装置前做出对应的手势动作。
例如，用户将大拇指竖起，其余四个手指握住。头戴显示设备可以是识别出目标用户的手势动作，并根据该手势动作确定输入指令为模式切换指令。
在另一些实施方式中，将所述关键点信息发送给指令分类模型；基于所述指令分类模型对所述关键点信息进行分类，以得到输入指令。
示例性的，指令分类模型可以通过根据手势样本集对深度神经网络进行训练至收敛得到。手势样本集包括若干关键点信息和所述关键点信息各自对应的指令标注。这些关键点信息对应于多种不同的手势。因此训练好的手势动作分类模型可以更准确的根据关键点信息识别目标用户的输入指令。
可以理解的，头戴显示设备可以不识别手势动作，而直接根据所述关键点信息识别输入指令。也可以不存储手势动作和输入指令的映射表。
在一些实施方式中，若根据所述关键点信息识别出的输入指令为模式切换指令，则根据所述模式切换指令开启或关闭肢体交互模式。
所述肢体交互模式用于根据识别出的输入指令产生的控制指令执行任务操作。
示例性的，目标用户在想开启肢体交互模式，以控制头戴显示设备可以根据目标用户作出的手势动作进行交互时，可以作出模式切换指令对应的手势动作。从而头戴显示设备可以根据确定的模式切换指令开启肢体交互模式。
在开启肢体交互模式时，头戴显示设备获取目标用户的手势动作的关键点信息，并根据所述关键点信息识别输入指令。如果再次识别到输入指令为模式切换指令，则关闭肢体交互模式。关闭肢体交互模式后，如果识别的输入指令不是模式切换指令，则不响应该输入指令；如果识别的输入指令是模式切换指 令，则开启肢体交互模式。
在开启肢体交互模式时，头戴显示设备响应识别的输入指令。
S130、根据所述输入指令产生控制指令，以根据所述控制指令执行任务操作。
在一些实施方式中，不同的输入指令对应于不同的控制指令。
示例性的，头戴显示设备预先存储有输入指令和控制指令的映射表。该映射表包括若干输入指令和若干控制指令之间的对应关系。
示例性的，用户可以通过设置操作，将输入指令关联于用户指定的功能，如保存头戴显示设备显示的画面的截屏功能等。头戴显示设备可以根据用户的设置操作，调整输入指令和控制指令之间的对应关系，该控制指令对应于用户指定的功能。
从而头戴显示设备可以根据识别出的输入指令确定对应的控制指令。然后根据输入指令产生控制指令，以根据所述控制指令执行任务操作。
在另一些实施方式中，如图3所示，所述头戴显示设备显示交互操作界面。
示例性的，所述根据所述控制指令执行任务操作，包括：根据所述控制指令切换所述头戴显示设备显示的交互操作界面。
示例性的，当所述输入指令为切换至下一个交互操作界面的指令时，则根据当前显示的交互操作界面和该输入指令，生成用于控制头戴显示设备切换至下一个交互操作界面的控制指令，以使所述头戴显示设备切换至下一个交互操作界面。
例如，若输入指令为确认指令，则根据当前显示的交互操作界面和该输入指令，生成用于控制头戴显示设备切换至下一个交互操作界面的控制指令，以使所述头戴显示设备切换至下一个交互操作界面。
例如，若输入指令为返回指令，则根据当前显示的交互操作界面和该输入指令，生成用于控制头戴显示设备切换至上一个交互操作界面的控制指令，以使所述头戴显示设备返回至上一个交互操作界面。
示例性的，所述根据所述输入指令产生控制指令，包括：根据所述头戴显示设备当前显示的交互操作界面和所述输入指令确定操作控制信息；根据所述操作控制信息产生对应的控制指令。
示例性的，头戴显示设备显示某交互界面时，允许根据用户的手势动作保 存头戴显示设备显示的画面。当头戴显示设备显示该交互界面时，若根据关键点信息识别出的输入指令为确认指令，则可以根据该输入指令确定操作控制信息对应于截屏功能。根据该操作控制信息产生截屏功能对应的控制指令，以使头戴显示设备保存当前显示的画面。
示例性的，交互操作界面显示一个虚拟按钮或多个虚拟按钮，虚拟按键对应于不同的操作控制信息。
例如，某交互操作界面包括截屏功能对应的虚拟按键。当头戴显示设备显示该交互界面时，若根据关键点信息识别出的输入指令为确认指令，则可以根据该输入指令确定操作控制信息对应于截屏功能。根据该操作控制信息产生截屏功能对应的控制指令，以使头戴显示设备保存当前显示的画面。
示例性的，所述根据所述交互操作界面和所述输入指令确定操作控制信息，包括：确定所述交互操作界面中的功能按钮；根据所述功能按钮和所述输入指令确定操作控制信息。
在一些实施方式中，如图5所示，头戴显示设备通信连接于可移动平台，如无人飞行器。
示例性的，如图3所示，交互操作界面包括多个功能按钮，如对应于控制无人飞行器返航的返航按钮、对应于控制无人飞行器悬停的悬停按钮、对应于截屏功能的截屏按钮和菜单按钮。
例如，确定所述交互操作界面中用户选定的功能按钮，然后根据用户选定的功能按钮和所述输入指令确定操作控制信息。
示例性的，头戴显示设备根据目标用户的手势动作的关键点信息确定目标用户手部的预设部位在所述交互操作界面上的位置；根据所述交互操作界面上的所述位置确定所述功能按钮。
例如，头戴显示设备根据手势动作的关键点信息确定目标用户食指指尖在交互操作界面上的位置。具体的，根据用户食指指尖在戴显示设备搭载的图像获取装置拍摄的用户图像中的位置，确定食指指尖对应的光标在交互操作界面上的位置。
示例性的，用户可以通过左手或者右手的食指的移动来控制交互操作界面中的光标。当光标移动到交互操作界面中的某一功能按钮时，确定该功能按钮为可响应所述输入指令的功能按钮。头戴显示设备可以响应输入指令来触发功 能按钮的功能，如控制无人飞行器悬停或返航等。
例如，在某交互操作界面确定的功能按钮为对应于返航指令的功能按钮时，若根据关键点信息识别出的输入指令为确认指令，则可以根据该输入指令确定对应于返航的操作控制信息。根据该操作控制信息产生控制指令，以使头戴显示设备根据控制指令向无人飞行器发送返航指令，从而控制无人飞行器返航。
在一些实施方式中，所述根据所述控制指令执行任务操作，包括：调整所述头戴显示设备当前显示的交互操作界面中的菜单。
示例性的，如图3所示，交互操作界面包括菜单按钮。在交互操作界面确定的功能按钮为该菜单按钮时，若根据关键点信息识别出的输入指令为确认指令，则可以根据该输入指令确定对应于展开该菜单的操作控制信息。根据该操作控制信息产生控制指令，以使头戴显示设备在该交互操作显示界面显示该菜单对应的选项按钮，如选项一和选项二。
例如，头戴显示设备根据目标用户的手势动作的关键点信息确定目标用户手部的预设部位在所述交互操作界面上的位置；根据所述交互操作界面上的所述位置确定选项二对应的功能按钮。若之后根据关键点信息识别出的输入指令为确认指令，则可以根据该输入指令确定对应于选项二的操作控制信息。根据该操作控制信息产生控制指令，以使头戴显示设备实现选项二对应的操作。
在一些实施方式中，所述头戴显示设备通信连接于可移动平台。
示例性的，可移动平台为无人飞行器、无人驾驶车辆或无人驾驶船艇。无人飞行器例如可以为旋翼型无人机，例如四旋翼无人机、六旋翼无人机、八旋翼无人机，也可以是固定翼无人机。
示例性的，所述根据所述控制指令执行任务操作，包括：根据所述控制指令开始存储或者结束存储所述头戴显示设备从可移动平台获取的数据。
示例性的，头戴显示设备通过与可移动平台之间的通信链路从可移动平台获取数据，如可移动平台搭载的图像获取装置拍摄的图像等。头戴显示设备在产生某一控制指令后，根据该控制指令开始存储或者结束存储从所述可移动平台获取的数据。
示例性的，所述根据所述控制指令执行任务操作，包括：根据所述控制指令向可移动平台发送平台控制指令，以控制所述可移动平台。
示例性的，若根据输入指令产生对应于返航的控制指令，则所述头戴显示 设备根据该控制指令向所述可移动平台发送用于控制可移动平台返航的平台控制指令。若根据输入指令产生对应于加速移动的控制指令，则所述头戴显示设备根据该控制指令向所述可移动平台发送用于控制可移动平台加速移动的平台控制指令。
在一些实施方式中，控制方法应用于手表，手表包括图像获取装置，例如包括摄像头。
手表通过摄像头获取目标用户的手势动作的关键点信息，然后根据所述关键点信息识别输入指令，并根据所述输入指令产生控制指令，以根据所述控制指令执行任务操作。
可以理解的，手表也可以显示交互操作界面。手表也可以根据所述控制指令切换显示的交互操作界面，可以根据当前显示的交互操作界面和所述输入指令确定操作控制信息并根据所述操作控制信息产生对应的控制指令等。
可以理解的，手表也可以通信连接于可移动平台，手表可以响应输入指令来触发功能按钮的功能，如控制无人飞行器悬停或返航等。
在一些实施方式中，如图5所示，头戴显示设备连接于可移动平台，可移动平台搭载有图像获取装置。例如通过有线方式连接于可移动平台或者通过无线通信方式连接于可移动平台。
可移动平台自带图像获取装置或者通过通信接口连接图像获取装置。图像获取装置包括彩色摄像头、深度传感器、RGB-D摄像头中的至少一种。图像获取装置例如为独立的摄像头或者为手机、相机等电子设备。
示例性的，步骤S110获取目标用户的手势动作的关键点信息，包括：根据可移动平台搭载的图像获取装置拍摄的用户图像和所述头戴显示设备搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
从而，头戴显示设备可以根据头戴显示设备拍摄的用户图像和可移动平台拍摄的用户图像确定手势动作的关键点信息。从而可以获取在不同视角下目标用户的用户图像，从而确定不同视角下目标用户的手部图像。可以得到更丰富的手势动作的关键点信息，且可以防止手部关键点部分被遮挡导致的手势识别准确率降低。例如，可以根据头戴显示设备拍摄的用户图像和可移动平台拍摄的用户图像确定目标用户手心和手背上的关键点信息。
在一些实施方式中，如图6所示，头戴显示设备连接于手持控制装置，手 持控制装置搭载有图像获取装置。例如通过有线方式连接于手持控制装置或者通过无线通信方式连接于手持控制装置。
示例性的，头戴显示设备还可以根据手持控制装置发送的控制指令执行相应的操作或功能，如切换交互操作界面、确定交互操作界面中的功能按钮、开启或关闭肢体交互模式等。
手持控制装置自带图像获取装置或者通过通信接口连接图像获取装置。图像获取装置包括彩色摄像头、深度传感器、RGB-D摄像头中的至少一种。图像获取装置例如为独立的摄像头或者为手机、相机等电子设备。
示例性的，步骤S110获取目标用户的手势动作的关键点信息，包括：根据手持控制装置搭载的图像获取装置拍摄的用户图像和所述头戴显示设备搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
从而，头戴显示设备可以根据头戴显示设备拍摄的用户图像和手持控制装置拍摄的用户图像确定手势动作的关键点信息。从而可以获取在不同视角下目标用户的用户图像，从而确定不同视角下目标用户的手部图像。可以得到更丰富的手势动作的关键点信息，且可以防止手部关键点部分被遮挡导致的手势识别准确率降低。例如，可以根据头戴显示设备拍摄的用户图像和手持控制装置拍摄的用户图像确定目标用户手心和手背上的关键点信息。
可以理解的，控制方法应用于手表等可穿戴式设备时，手表等可穿戴式设备也可以通信连接于可移动平台和/或手持控制装置。
手表等可穿戴式设备也可以根据可穿戴式设备拍摄的用户图像以及手持控制装置和/或可移动平台拍摄的用户图像确定手势动作的关键点信息。
在一些实施方式中，如图7所示，步骤S110中获取目标用户的手势动作的关键点信息，包括步骤S111至步骤S114。
S111、获取头戴显示设备搭载的图像获取装置拍摄的用户图像。
S112、根据所述头戴显示设备搭载的图像获取装置拍摄的用户图像判断目标用户的手势动作的关键点信息是否被遮挡。
示例性的，根据头戴显示设备搭载的图像获取装置拍摄的用户图像确定手部图像中的关键点，如果关键点的数量小于预设阈值，如22，则判定手势动作的关键点信息被遮挡；如果关键点的数量不小于预设阈值，则判定手势动作的关键点信息未被遮挡。
如果判定手势动作的关键点信息未被遮挡，则根据所述头戴显示设备搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
S113、若所述关键点信息被遮挡，获取所述手持控制装置搭载的图像获取装置拍摄的用户图像和/或获取所述可移动平台搭载的图像获取装置拍摄的用户图像。
如图5、图6或图8所示，头戴显示设备连接于可移动平台和/或手持控制装置。可移动平台和/或手持控制装置各自搭载有图像获取装置。
如果判定头戴显示设备搭载的图像获取装置拍摄的用户图像中手势动作的关键点信息被遮挡，则请求可移动平台和/或手持控制装置发送可移动平台和/或手持控制装置搭载的图像获取装置拍摄的用户图像。
S114、根据所述头戴显示设备搭载的图像获取装置拍摄的用户图像，以及所述手持控制装置搭载的图像获取装置拍摄的用户图像和/或所述可移动平台搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
从而，头戴显示设备可以根据头戴显示设备拍摄的用户图像以及可移动平台和/或手持控制装置拍摄的用户图像确定手势动作的关键点信息。从而可以获取在不同视角下目标用户的用户图像，从而确定不同视角下目标用户的手部图像。可以得到更丰富的手势动作的关键点信息，且可以防止手部关键点部分被遮挡导致的手势识别准确率降低。例如，可以确定目标用户手心和手背上的关键点信息。
在一些实施方式中，步骤S110中获取目标用户的手势动作的关键点信息，包括：获取可移动平台搭载的图像获取装置拍摄的用户图像；根据所述可移动平台搭载的图像获取装置拍摄的用户图像判断目标用户的手势动作的关键点信息是否被遮挡。如果判定手势动作的关键点信息未被遮挡，则根据所述可移动平台搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。若所述关键点信息被遮挡，获取可穿戴设备，例如头戴显示设备、智能手表等搭载的图像获取装置拍摄的用户图像。之后根据所述可穿戴设备搭载的图像获取装置拍摄的用户图像，以及所述可移动平台搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
在一些实施方式中，可移动平台根据手势动作的关键点信息获得所述手势的识别结果，并根据识别结果执行相应的操作，如检测可移动平台搭载的图像 获取装置拍摄的图像中的拍摄目标，对拍摄目标进行拍摄等。
可移动平台或者可穿戴设备可以根据可穿戴设备拍摄的用户图像以及可移动平台拍摄的用户图像确定手势动作的关键点信息。从而可以获取在不同视角下目标用户的用户图像，从而确定不同视角下目标用户的手部图像。可以得到更丰富的手势动作的关键点信息，且可以防止手部关键点部分被遮挡导致的手势识别准确率降低。例如，可以确定目标用户手心和手背上的关键点信息。
在一些实施方式中，如图6所示，目标用户手持手持控制装置，并佩戴头戴显示设备；可移动平台可以根据手持控制装置和/或头戴显示设备发送的指令执行相应的动作或功能。用户可以将手稍微离开手持控制装置，作出相应的手势动作，就可以向头戴显示设备输入手势动作相应的指令，以使头戴显示设备根据输入的指令产生控制指令，根据所述控制指令执行任务操作。从而，用户无需将手抬到头上佩戴的头戴显示设备的触控板上进行控制，更加省力。而且基于头戴显示设备显示的交互操作界面，用户可以直接用手在二维上进行菜单的选择和触发，如即可以控制界面上的光标左右或上下移动，还可以控制光标向坐上、右下等不同的方向移动，操作更加快捷，体验更加友好。
本说明书实施例提供的头戴显示设备的控制方法，通过获取目标用户的手势动作的关键点信息，并根据关键点信息识别输入指令，以及根据输入指令产生控制指令，以根据控制指令执行任务操作；实现用户可以快捷的使用手悬空作出手势动作控制头戴显示设备，且通过关键点信息识别手势动作对应的输入指令，可以排除会干扰手势识别的信息，识别速度更快且更准确，从而方便用户更快捷和更准确的控制头戴显示设备。
可以理解的，可以通过头戴显示设备拍摄的用户图像以及可移动平台和/或手持控制装置拍摄的用户图像确定手势动作的关键点信息，得到更丰富的手势动作的关键点信息，且可以防止手部关键点部分被遮挡导致的手势识别准确率降低。
可以理解的，用户可以通过特定的手势来“按下”交互操作界面中的虚拟按键，触发对应按键功能。根据手部关键点得到用户的食指指尖的位置确定交互操作界面中光标的位置，进而可以根据食指指尖的移动来控制菜单；以及将手部关键点信息输入到手势动作分类模型，识别用户的手势，当用户做出特定的手势如“剪刀手”时，可以“按下”光标所处位置的虚拟按钮，控制头戴显 示设备或者通过头戴显示设备控制器连接的可移动平台和或手持控制装置。
请结合上述实施例参阅图9，图9是本说明书一实施例提供的可穿戴设备600的示意性框图。
具体的，可穿戴设备600可以为虚拟现实(VR，virtual reality)显示设备或第一人称视角(FPV，first person view)显示设备等。可穿戴设备600例如可以为眼镜式显示设备或者头盔式显示设备等。
该可穿戴设备600包括处理器601和存储器602，处理器601和存储器602通过总线603连接，该总线603比如为I2C(Inter-integrated Circuit)总线。
具体地，处理器601可以是微控制单元(Micro-controller Unit，MCU)、中央处理单元(Central Processing Unit，CPU)或数字信号处理器(Digital Signal Processor，DSP)等。
具体地，存储器602可以是Flash芯片、只读存储器(ROM，Read-Only Memory)磁盘、光盘、U盘或移动硬盘等。
具体的，可穿戴设备600自带图像获取装置604或者通过通信接口连接图像获取装置604。例如，图像获取装置604位于可穿戴设备600的下侧、上侧、左侧或右侧。
其中，所述处理器601用于运行存储在存储器602中的计算机程序，并在执行所述计算机程序时实现前述的控制方法。
本说明书实施例提供的可穿戴设备的具体原理和实现方式均与前述实施例的控制方法类似，此处不再赘述。
请结合上述实施例参阅图5，图5是本说明书一实施例提供的可移动平台控制系统的示意图。
如图5所示，可移动平台控制系统包括可移动平台和可穿戴设备。
其中，可移动平台，包括图像获取装置，用于将图像获取装置拍摄的图像发送给可穿戴设备；可穿戴设备，用于显示所述可移动平台发送的图像。
在一些实施方式中，如图8所示，可移动平台控制系统还包括：手持控制装置，用于向所述可移动平台发送遥控指令。所述可移动平台还用于根据所述遥控指令移动。
本说明书实施例提供的可移动平台控制系统的具体原理和实现方式均与前 述实施例的控制方法类似，此处不再赘述。
请结合上述实施例参阅图10，图10是本说明书一实施例提供的识别手势的方法的流程示意图。
识别手势的方法可以应用于搭载图像获取装置的可穿戴设备、可移动平台等。
可穿戴设备例如可以为头戴显示设备、智能手表、上衣、腰带、护带等；头戴显示设备可以为虚拟现实(VR，virtual reality)显示设备或第一人称视角(FPV，first person view)显示设备。头戴显示设备例如可以为眼镜式显示设备或者头盔式显示设备等。
可移动平台例如可以为搭载图像获取装置的无人机、手持云台、移动机器人、车辆等。
如图10所示，识别手势的方法包括步骤S210至步骤S250。
S210、获得所述手势的第一图像。
示例性的，所述第一图像通过可穿戴设备搭载的图像获取装置获得。
示例性的，获取可穿戴设备搭载的图像获取装置拍摄的用户图像。
示例性的，所述可穿戴设备是具有图像获取装置的头盔、手表、眼镜、上衣、腰带、护带。
S220、根据所述第一图像获得所述手势的关键点信息。
示例性的，在全图范围内进行手部检测，输出手部的具体位置。根据手部的具体位置从摄像头拍摄的图像中裁剪出手部区域的图片。再将手部区域的图片输入到手部关键点检测神经网络得到手势的关键点信息，如图片中手部关键点的数目、位置等信息。
S230、当根据所述关键点信息无法判断所述手势类型时，获得所述手势的第二图像。
示例性的，如果手势的关键点信息中关键点的数量小于预设阈值，如22，则判定手势动作的关键点信息被遮挡，判定根据所述关键点信息无法判断所述手势类型。
如果关键点的数量不小于预设阈值，则判定手势动作的关键点信息未被遮挡，可以根据关键点信息获得所述手势的识别结果。
示例性的，所述第二图像通过所述可穿戴设备搭载的另一图像获取装置获 得。
在一些实施例中，所述头戴显示设备在不同位置搭载至少两个摄像头。头戴显示设备根据在不同位置搭载的至少两个摄像头拍摄的用户图像确定所述手势动作的关键点信息。从而可以在不同视角下获取目标用户的用户图像，从而确定不同视角下目标用户的手部图像。可以得到更丰富的手势动作的关键点信息。
S240、根据所述第二图像更新所述手势的关键点信息。
示例性的，根据多个摄像头的图像可以分别获取手心和手背上的关键点信息，防止手部关键点部分被遮挡导致的手势识别准确率降低。
示例性的，将第一图像中的关键点的位置信息和第二图像中的关键点的位置信息一起存储为所述手势的关键点信息。
S250、根据所述更新后的关键点信息获得所述手势的识别结果。
根据手势动作的关键点信息可以确定手势动作的种类。例如，将所述关键点信息发送给手势动作分类模型；基于所述手势动作分类模型对所述关键点信息进行分类，得到所述手势的识别结果。
在另一些实施方式中，所述第一图像通过可移动平台搭载的图像获取装置获得。
示例性的，所述可移动平台是带有图像获取装置的无人机、手持云台、移动机器人、车辆。
示例性的，获取可移动平台搭载的图像获取装置拍摄的用户图像；根据所述可移动平台搭载的图像获取装置拍摄的用户图像判断目标用户的手势动作的关键点信息是否被遮挡。如果判定手势动作的关键点信息未被遮挡，则根据所述可移动平台搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。若所述关键点信息被遮挡，获取可穿戴设备，例如头戴显示设备、智能手表等搭载的图像获取装置拍摄的用户图像。之后根据所述可穿戴设备搭载的图像获取装置拍摄的用户图像，以及所述可移动平台搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
示例性的，所述第二图像通过可穿戴设备搭载的图像获取装置获得。
本说明书实施例提供的识别手势的方法，通过在根据第一图像的关键点信息无法判断手势类型时，获得手势的第二图像，并根据第二图像更新手势的关 键点信息，以根据更新后的关键点信息获得所述手势的识别结果。可以得到更丰富的手势动作的关键点信息，且可以防止手部关键点部分被遮挡导致的手势识别准确率降低。
请结合上述实施例参阅图11，图11是本说明书一实施例提供的可穿戴设备70的示意性框图。
如图11所示，可穿戴设备70包括存储器71和处理器72；所述存储器71用于存储计算机程序；所述处理器72，用于执行所述计算机程序并在执行所述计算机程序时，实现前述的识别手势的方法。
示例性的，所述可穿戴设备包括一下至少一种：头戴显示设备、智能手表、上衣、腰带、护带。
本说明书实施例提供的可穿戴设备的具体原理和实现方式均与前述实施例的控制方法类似，此处不再赘述。
本说明书的实施例中还提供一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序中包括程序指令，所述处理器执行所述程序指令，实现上述实施例提供的控制方法的步骤和/或识别手势的方法的步骤。
示例性的，所述计算机可读存储介质可以是前述任一实施例所述的可穿戴设备的内部存储单元，例如所述可穿戴设备的硬盘或内存。所述计算机可读存储介质也可以是所述可穿戴设备的外部存储设备，例如所述可穿戴设备上配备的插接式硬盘，智能存储卡(Smart Media Card，SMC)，安全数字(Secure Digital，SD)卡，闪存卡(Flash Card)等。
本说明书上述实施例提供的可穿戴设备、识别手势的方法、可移动平台控制系统、计算机可读存储介质，通过获取目标用户的手势动作的关键点信息，并根据关键点信息识别输入指令，以及根据输入指令产生控制指令，以根据控制指令执行任务操作；实现用户可以快捷的使用手悬空作出手势动作控制可穿戴设备，且通过关键点信息识别手势动作对应的输入指令，可以排除会干扰手势识别的信息，识别速度更快且更准确，从而方便用户更快捷和更准确的控制可穿戴设备。
可以理解的，可以通过可穿戴设备拍摄的用户图像以及可移动平台和/或手持控制装置拍摄的用户图像确定手势动作的关键点信息，得到更丰富的手势动 作的关键点信息，且可以防止手部关键点部分被遮挡导致的手势识别准确率降低。
可以理解的，用户可以通过特定的手势来“按下”交互操作界面中的虚拟按键，触发对应按键功能。根据手部关键点得到用户的食指指尖的位置确定交互操作界面中光标的位置，进而可以根据食指指尖的移动来控制菜单；以及将手部关键点信息输入到手势动作分类模型，识别用户的手势，当用户做出特定的手势如“剪刀手”时，可以“按下”光标所处位置的虚拟按钮，控制可穿戴设备或者通过可穿戴设备控制器连接的可移动平台和或手持控制装置。
应当理解，在此本说明书中所使用的术语仅仅是出于描述特定实施例的目的而并不意在限制本说明书。
还应当理解，在本说明书和所附权利要求书中使用的术语“和/或”是指相关联列出的项中的一个或多个的任何组合以及所有可能组合，并且包括这些组合。
以上所述，仅为本说明书的具体实施方式，但本说明书的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本说明书揭露的技术范围内，可轻易想到各种等效的修改或替换，这些修改或替换都应涵盖在本说明书的保护范围之内。因此，本说明书的保护范围应以权利要求的保护范围为准。

Claims (50)

  1. 一种可穿戴设备的控制方法，其特征在于，包括：
    获取目标用户的手势动作的关键点信息；
    根据所述关键点信息识别输入指令；
    根据所述输入指令产生控制指令，以通过所述控制指令执行任务操作。
  2. 根据权利要求1所述的控制方法，其特征在于，所述获取目标用户的手势动作的关键点信息，包括：
    获取可穿戴设备搭载的图像获取装置拍摄的用户图像；
    根据所述可穿戴设备搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
  3. 根据权利要求2所述的控制方法，其特征在于，所述根据所述可穿戴设备搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息，包括：
    根据所述可穿戴设备搭载的至少两个图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
  4. 根据权利要求2所述的控制方法，其特征在于，还包括：
    根据手持控制装置搭载的图像获取装置拍摄的用户图像和所述可穿戴设备搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
  5. 根据权利要求2所述的控制方法，其特征在于，还包括：
    根据可移动平台搭载的图像获取装置拍摄的用户图像和所述可穿戴设备搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
  6. 根据权利要求2所述的控制方法，其特征在于，还包括：
    根据所述可穿戴设备搭载的图像获取装置拍摄的用户图像判断目标用户的手势动作的关键点信息是否被遮挡；
    若所述关键点信息被遮挡，获取手持控制装置搭载的图像获取装置拍摄的用户图像和/或获取可移动平台搭载的图像获取装置拍摄的用户图像；
    根据所述可穿戴设备搭载的图像获取装置拍摄的用户图像，以及所述手持控制装置搭载的图像获取装置拍摄的用户图像和/或所述可移动平台搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
  7. 根据权利要求2所述的控制方法，其特征在于，所述获取目标用户的手势动作的关键点信息，包括：
    识别所述用户图像中的手部区域，并获取所述手部区域中的关键点信息。
  8. 根据权利要求1所述的控制方法，其特征在于，所述根据所述关键点信息识别输入指令，包括：
    将所述关键点信息发送给手势动作分类模型；
    基于所述手势动作分类模型对所述关键点信息进行分类，以得到输入指令。
  9. 根据权利要求1所述的控制方法，其特征在于，所述输入指令包括确定指令、返回指令、模式切换指令中的至少一种。
  10. 根据权利要求9所述的控制方法，其特征在于，还包括：
    若根据所述关键点信息识别出的输入指令为模式切换指令，根据所述模式切换指令开启或关闭肢体交互模式；
    所述肢体交互模式用于根据识别出的输入指令产生的控制指令执行任务操作。
  11. 根据权利要求1所述的控制方法，其特征在于，所述根据所述输入指令产生控制指令，包括：
    根据所述可穿戴设备当前显示的交互操作界面和所述输入指令确定操作控制信息；
    根据所述操作控制信息产生对应的控制指令。
  12. 根据权利要求11所述的控制方法，其特征在于，所述根据所述交互操作界面和所述输入指令确定操作控制信息，包括：
    确定所述交互操作界面中的功能按钮；
    根据所述功能按钮和所述输入指令确定操作控制信息。
  13. 根据权利要求12所述的控制方法，其特征在于，所述确定所述交互操作界面中的功能按钮，包括：
    根据所述关键点信息确定目标用户手部的预设部位在所述交互操作界面上的位置；
    根据所述交互操作界面上的所述位置确定所述功能按钮。
  14. 根据权利要求11所述的控制方法，其特征在于，所述通过所述控制指令执行任务操作，包括：
    根据所述控制指令切换所述可穿戴设备显示的交互操作界面，或者调整所述可穿戴设备当前显示的交互操作界面中的菜单。
  15. 根据权利要求1所述的控制方法，其特征在于，所述通过所述控制指令执行任务操作，包括：
    根据所述控制指令开始存储或者结束存储所述可穿戴设备从可移动平台获取的数据。
  16. 根据权利要求1所述的控制方法，其特征在于，所述通过所述控制指令执行任务操作，包括：
    根据所述控制指令向可移动平台发送平台控制指令，以控制所述可移动平台。
  17. 根据权利要求5所述的控制方法，其特征在于，所述可移动平台为无人飞行器、无人驾驶车辆或无人驾驶船艇。
  18. 根据权利要求1所述的控制方法，其特征在于，所述可穿戴设备为虚拟现实显示设备或第一人称视角显示设备。
  19. 根据权利要求2所述的控制方法，其特征在于，所述可穿戴设备搭载的图像获取装置包括彩色摄像头、深度传感器、RGB-D摄像头中的至少一种。
  20. 一种可穿戴设备，其特征在于，包括存储器和处理器；
    所述存储器用于存储计算机程序；
    所述处理器，用于执行所述计算机程序并在执行所述计算机程序时，实现如下步骤：
    获取目标用户的手势动作的关键点信息；
    根据所述关键点信息识别输入指令；
    根据所述输入指令产生控制指令，以根据所述控制指令执行任务操作。
  21. 根据权利要求20所述的可穿戴设备，其特征在于，所述获取目标用户的手势动作的关键点信息，包括：
    获取可穿戴设备搭载的图像获取装置拍摄的用户图像；
    根据所述可穿戴设备搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
  22. 根据权利要求21所述的可穿戴设备，其特征在于，所述根据所述可穿戴设备搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息， 包括：
    根据所述可穿戴设备搭载的至少两个图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
  23. 根据权利要求21所述的可穿戴设备，其特征在于，所述处理器还实现：
    根据手持控制装置搭载的图像获取装置拍摄的用户图像和所述可穿戴设备搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
  24. 根据权利要求21所述的可穿戴设备，其特征在于，所述处理器还实现：
    根据可移动平台搭载的图像获取装置拍摄的用户图像和所述可穿戴设备搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
  25. 根据权利要求21所述的可穿戴设备，其特征在于，所述处理器还实现：
    根据所述可穿戴设备搭载的图像获取装置拍摄的用户图像判断目标用户的手势动作的关键点信息是否被遮挡；
    若所述关键点信息被遮挡，获取手持控制装置搭载的图像获取装置拍摄的用户图像和/或获取可移动平台搭载的图像获取装置拍摄的用户图像；
    根据所述可穿戴设备搭载的图像获取装置拍摄的用户图像，以及所述手持控制装置搭载的图像获取装置拍摄的用户图像和/或所述可移动平台搭载的图像获取装置拍摄的用户图像确定所述手势动作的关键点信息。
  26. 根据权利要求21所述的可穿戴设备，其特征在于，所述获取目标用户的手势动作的关键点信息，包括：
    识别所述用户图像中的手部区域，并获取所述手部区域中的关键点信息。
  27. 根据权利要求20所述的可穿戴设备，其特征在于，所述根据所述关键点信息识别输入指令，包括：
    将所述关键点信息发送给手势动作分类模型；
    基于所述手势动作分类模型对所述关键点信息进行分类，以得到输入指令。
  28. 根据权利要求20所述的可穿戴设备，其特征在于，所述输入指令包括确定指令、返回指令、模式切换指令中的至少一种。
  29. 根据权利要求28所述的可穿戴设备，其特征在于，还包括：
    若根据所述关键点信息识别出的输入指令为模式切换指令，根据所述模式切换指令开启或关闭肢体交互模式；
    所述肢体交互模式用于根据识别出的输入指令产生的控制指令执行任务操 作。
  30. 根据权利要求20所述的可穿戴设备，其特征在于，所述根据所述输入指令产生控制指令，包括：
    根据所述可穿戴设备当前显示的交互操作界面和所述输入指令确定操作控制信息；
    根据所述操作控制信息产生对应的控制指令。
  31. 根据权利要求30所述的可穿戴设备，其特征在于，所述根据所述交互操作界面和所述输入指令确定操作控制信息，包括：
    确定所述交互操作界面中的功能按钮；
    根据所述功能按钮和所述输入指令确定操作控制信息。
  32. 根据权利要求31所述的可穿戴设备，其特征在于，所述确定所述交互操作界面中的功能按钮，包括：
    根据所述关键点信息确定目标用户手部的预设部位在所述交互操作界面上的位置；
    根据所述交互操作界面上的所述位置确定所述功能按钮。
  33. 根据权利要求30所述的可穿戴设备，其特征在于，所述根据所述控制指令执行任务操作，包括：
    根据所述控制指令切换所述可穿戴设备显示的交互操作界面，或者调整所述可穿戴设备当前显示的交互操作界面中的菜单。
  34. 根据权利要求20所述的可穿戴设备，其特征在于，所述根据所述控制指令执行任务操作，包括：
    根据所述控制指令开始存储或者结束存储所述可穿戴设备从可移动平台获取的数据。
  35. 根据权利要求20所述的可穿戴设备，其特征在于，所述根据所述控制指令执行任务操作，包括：
    根据所述控制指令向可移动平台发送平台控制指令，以控制所述可移动平台。
  36. 根据权利要求24所述的可穿戴设备，其特征在于，所述可移动平台为无人飞行器、无人驾驶车辆或无人驾驶船艇。
  37. 根据权利要求20所述的可穿戴设备，其特征在于，所述可穿戴设备为 虚拟现实显示设备或第一人称视角显示设备。
  38. 根据权利要求21所述的可穿戴设备，其特征在于，所述可穿戴设备搭载的图像获取装置包括彩色摄像头、深度传感器、RGB-D摄像头中的至少一种。
  39. 一种可移动平台控制系统，其特征在于，包括：
    可移动平台，包括图像获取装置，用于将图像获取装置拍摄的图像发送给可穿戴设备；
    如权利要求20-38中任一项所述的可穿戴设备，用于显示所述可移动平台发送的图像。
  40. 根据权利要求39所述的控制系统，其特征在于，还包括：
    手持控制装置，用于向所述可移动平台发送遥控指令；
    所述可移动平台还用于根据所述遥控指令移动。
  41. 一种识别手势的方法，其特征在于，所述方法包括：
    获得所述手势的第一图像；
    根据所述第一图像获得所述手势的关键点信息；
    当根据所述关键点信息无法判断所述手势类型时，获得所述手势的第二图像；
    根据所述第二图像更新所述手势的关键点信息；
    根据所述更新后的关键点信息获得所述手势的识别结果。
  42. 根据权利要求41所述的方法，其特征在于，所述第一图像通过可穿戴设备搭载的图像获取装置获得。
  43. 根据权利要求42所述的方法，其特征在于，所述第二图像通过所述可穿戴设备搭载的另一图像获取装置获得。
  44. 根据权利要求41所述的方法，其特征在于，所述第一图像通过可移动平台搭载的图像获取装置获得。
  45. 根据权利要求44所述的方法，其特征在于，所述第二图像通过可穿戴设备搭载的图像获取装置获得。
  46. 根据权利要求42、43、45中任一项所述的方法，其特征在于，所述可穿戴设备是具有图像获取装置的头盔、手表、眼镜、上衣、腰带、护带。
  47. 根据权利要求44所述的方法，其特征在于，所述可移动平台是带有图像获取装置的无人机、手持云台、移动机器人、车辆。
  48. 一种可穿戴设备，其特征在于，包括存储器和处理器；
    所述存储器用于存储计算机程序；
    所述处理器，用于执行所述计算机程序并在执行所述计算机程序时，实现如权利要求41-43、45中任一项所述的方法。
  49. 根据权利要求48所述的可穿戴设备，其特征在于，所述可穿戴设备包括一下至少一种：头戴显示设备、智能手表、上衣、腰带、护带。
  50. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行时使所述处理器实现：
    如权利要求1-19中任一项所述的可穿戴设备的控制方法；和/或
    如权利要求41-47中任一项所述的识别手势的方法。
PCT/CN2019/103440 2019-08-29 2019-08-29 可穿戴设备及其控制方法、识别手势的方法和控制系统 WO2021035646A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
PCT/CN2019/103440 WO2021035646A1 (zh) 2019-08-29 2019-08-29 可穿戴设备及其控制方法、识别手势的方法和控制系统
CN201980033213.5A CN112154402A (zh) 2019-08-29 2019-08-29 可穿戴设备及其控制方法、识别手势的方法和控制系统
US17/472,761 US11782514B2 (en) 2019-08-29 2021-09-13 Wearable device and control method thereof, gesture recognition method, and control system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/103440 WO2021035646A1 (zh) 2019-08-29 2019-08-29 可穿戴设备及其控制方法、识别手势的方法和控制系统

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/472,761 Continuation US11782514B2 (en) 2019-08-29 2021-09-13 Wearable device and control method thereof, gesture recognition method, and control system

Publications (1)

Publication Number Publication Date
WO2021035646A1 true WO2021035646A1 (zh) 2021-03-04

Family

ID=73891948

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/103440 WO2021035646A1 (zh) 2019-08-29 2019-08-29 可穿戴设备及其控制方法、识别手势的方法和控制系统

Country Status (3)

Country Link
US (1) US11782514B2 (zh)
CN (1) CN112154402A (zh)
WO (1) WO2021035646A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114839782A (zh) * 2022-06-07 2022-08-02 上汽大众汽车有限公司 一种用于车辆控制与信息显示的车载增强显示系统

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022141187A1 (en) * 2020-12-30 2022-07-07 SZ DJI Technology Co., Ltd. Systems and methods for controlling an unmanned aerial vehicle using a body-attached remote control
CN112799507B (zh) * 2021-01-15 2022-01-04 北京航空航天大学 人体虚拟模型展示方法、装置、电子设备及存储介质
US11769134B2 (en) 2021-03-22 2023-09-26 International Business Machines Corporation Multi-user interactive ad shopping using wearable device gestures
US11726553B2 (en) 2021-07-20 2023-08-15 Sony Interactive Entertainment LLC Movement-based navigation
US11786816B2 (en) 2021-07-30 2023-10-17 Sony Interactive Entertainment LLC Sharing movement data
US20230051703A1 (en) * 2021-08-16 2023-02-16 Sony Interactive Entertainment LLC Gesture-Based Skill Search
CN114706486A (zh) * 2022-04-26 2022-07-05 四川大学 基于手势识别的混合现实工业控制方法及装置

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TW201544993A (zh) * 2014-05-28 2015-12-01 Pegatron Corp 手勢控制之方法、手勢控制模組及其具有手勢控制模組之穿戴式裝置
CN106020227A (zh) * 2016-08-12 2016-10-12 北京奇虎科技有限公司 无人机的控制方法、装置
KR101724108B1 (ko) * 2015-10-26 2017-04-06 재단법인대구경북과학기술원 손 모양 및 제스처에 의한 기기 제어 방법 및 그에 의한 제어 장치
CN106843669A (zh) * 2016-12-06 2017-06-13 北京小度信息科技有限公司 应用界面操作方法和装置
CN108874142A (zh) * 2018-06-26 2018-11-23 哈尔滨拓博科技有限公司 一种基于手势的无线智能控制装置及其控制方法
CN109190461A (zh) * 2018-07-23 2019-01-11 中南民族大学 一种基于手势关键点的动态手势识别方法和系统

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10509466B1 (en) * 2011-05-11 2019-12-17 Snap Inc. Headwear with computer and optical element for use therewith and systems utilizing same
KR20160138806A (ko) * 2015-05-26 2016-12-06 엘지전자 주식회사 글래스타입 단말기 및 그 제어방법
US11404056B1 (en) * 2016-06-30 2022-08-02 Snap Inc. Remoteless control of drone behavior
KR20180017674A (ko) * 2016-08-10 2018-02-21 엘지전자 주식회사 이동 단말기 및 그 제어 방법

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TW201544993A (zh) * 2014-05-28 2015-12-01 Pegatron Corp 手勢控制之方法、手勢控制模組及其具有手勢控制模組之穿戴式裝置
KR101724108B1 (ko) * 2015-10-26 2017-04-06 재단법인대구경북과학기술원 손 모양 및 제스처에 의한 기기 제어 방법 및 그에 의한 제어 장치
CN106020227A (zh) * 2016-08-12 2016-10-12 北京奇虎科技有限公司 无人机的控制方法、装置
CN106843669A (zh) * 2016-12-06 2017-06-13 北京小度信息科技有限公司 应用界面操作方法和装置
CN108874142A (zh) * 2018-06-26 2018-11-23 哈尔滨拓博科技有限公司 一种基于手势的无线智能控制装置及其控制方法
CN109190461A (zh) * 2018-07-23 2019-01-11 中南民族大学 一种基于手势关键点的动态手势识别方法和系统

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114839782A (zh) * 2022-06-07 2022-08-02 上汽大众汽车有限公司 一种用于车辆控制与信息显示的车载增强显示系统
CN114839782B (zh) * 2022-06-07 2023-08-18 上汽大众汽车有限公司 一种用于车辆控制与信息显示的车载增强显示系统

Also Published As

Publication number Publication date
US20210405763A1 (en) 2021-12-30
US11782514B2 (en) 2023-10-10
CN112154402A (zh) 2020-12-29

Similar Documents

Publication Publication Date Title
WO2021035646A1 (zh) 可穿戴设备及其控制方法、识别手势的方法和控制系统
KR101844390B1 (ko) 사용자 인터페이스 제어를 위한 시스템 및 기법
WO2018133593A1 (zh) 智能终端的控制方法和装置
JP2023169411A (ja) 情報処理装置、情報処理装置の制御方法、及びプログラム
JP6390799B2 (ja) 入力装置、入力方法、及びプログラム
US10564712B2 (en) Information processing device, information processing method, and program
US10372223B2 (en) Method for providing user commands to an electronic processor and related processor program and electronic circuit
CN109069920B (zh) 手持式控制器、跟踪定位方法以及系统
US10621766B2 (en) Character input method and device using a background image portion as a control region
US20180253149A1 (en) Information processing system, information processing apparatus, control method, and program
CN111766937A (zh) 虚拟内容的交互方法、装置、终端设备及存储介质
CN109947243B (zh) 基于触摸手检测的智能电子设备手势捕获与识别技术
WO2015104919A1 (ja) ジェスチャー認識装置、操作入力装置およびジェスチャー認識方法
CN112783318A (zh) 人机交互系统和人机交互方法
KR101233793B1 (ko) 손 동작 인식을 이용한 가상 마우스 구동 방법
US11500453B2 (en) Information processing apparatus
JPWO2017134732A1 (ja) 入力装置、入力支援方法および入力支援プログラム
CN109960406B (zh) 基于双手手指之间动作的智能电子设备手势捕获与识别技术
WO2023227072A1 (zh) 在虚拟现实场景中确定虚拟光标方法、装置、设备和介质
JP5558899B2 (ja) 情報処理装置、その処理方法及びプログラム
JP6971788B2 (ja) 画面表示制御方法および画面表示制御システム
CN112818825B (zh) 工作状态确定方法和装置
CN109993059B (zh) 智能电子设备上基于单摄像头的双目视觉与物体识别技术
WO2018150757A1 (ja) 情報処理システム、情報処理方法、およびプログラム
WO2021128414A1 (zh) 一种可穿戴设备及其输入方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19943009

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19943009

Country of ref document: EP

Kind code of ref document: A1