WO2017118075A1 - 人机交互系统、方法及装置 - Google Patents

人机交互系统、方法及装置 Download PDF

Info

Publication number
WO2017118075A1
WO2017118075A1 PCT/CN2016/098568 CN2016098568W WO2017118075A1 WO 2017118075 A1 WO2017118075 A1 WO 2017118075A1 CN 2016098568 W CN2016098568 W CN 2016098568W WO 2017118075 A1 WO2017118075 A1 WO 2017118075A1
Authority
WO
WIPO (PCT)
Prior art keywords
gesture
image
current
category
display device
Prior art date
Application number
PCT/CN2016/098568
Other languages
English (en)
French (fr)
Inventor
卢永春
Original Assignee
京东方科技集团股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 京东方科技集团股份有限公司 filed Critical 京东方科技集团股份有限公司
Priority to US15/519,689 priority Critical patent/US10585488B2/en
Publication of WO2017118075A1 publication Critical patent/WO2017118075A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16—Constructional details or arrangements
    • G06F1/1613—Constructional details or arrangements for portable computers
    • G06F1/163—Wearable computers, e.g. on a belt
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304—Detection arrangements using opto-electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G06F3/0425—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type

Definitions

  • the present disclosure relates to the field of human-computer interaction technologies, and in particular, to a human-computer interaction system, method, and apparatus.
  • the user in conjunction with virtual reality technology for human-computer interaction activities, the user needs to be in direct contact with the computer or use an entity input device to issue operational instructions to the computer. For example, the user performs a touch operation on the computer to implement various functions of the computer, or the computer is connected with a mouse and a keyboard, and the user uses the mouse and the keyboard to issue an operation instruction to the computer, thereby realizing various functions of the computer.
  • the manner in which the user needs to use the physical physical device to operate the computer is complicated, the user's operation of the human-computer interaction activity of the virtual reality technology is relatively low.
  • the present disclosure provides a human-computer interaction system, method, and apparatus for improving user convenience of performing human-computer interaction activities of virtual reality technology.
  • the present disclosure provides a human-machine interaction system, including: a wearable device and a display device;
  • the wearable device includes an image acquisition module, a memory, a processor, an image projection module, and an information sending interface;
  • the image acquisition module is connected to the processor, and is configured to collect a current gesture image of the user and a display image of the display device;
  • the processor is connected to the memory, configured to match the current gesture image with a gesture in a pre-stored gesture database in the memory, determine a gesture category of the matched gesture, and determine a control instruction corresponding to the gesture category Generating, according to the control instruction, a virtual tool image corresponding to the control instruction;
  • the image projection module is coupled to the processor for projecting a virtual image including the virtual tool image and the display image;
  • the information sending interface is connected to the processor, and configured to send the control instruction to the display device;
  • the display device is configured to execute the control instruction upon receiving the control instruction.
  • the present disclosure provides a human-computer interaction method, including:
  • the wearable device collects a current gesture image of the user and a display image of the display device
  • the present disclosure provides a human-machine interaction apparatus, including:
  • An acquisition module configured to collect a current gesture image of the user and a display image of the display device
  • a determining module configured to match the current gesture image with a gesture in a pre-stored gesture database, determine a gesture category of the matched gesture, and determine a control instruction corresponding to the gesture category;
  • a generating module configured to generate, according to the control instruction, a virtual tool image corresponding to the control instruction
  • a projection module configured to project a virtual image including the virtual tool image and a display image of the display device
  • a sending module configured to send the control instruction to the display device, so that the display device executes the control instruction.
  • the human-computer interaction system, method and device provided by the present disclosure collect the current gesture image of the user and the display image of the display device, match the current gesture image with the gesture in the gesture database, determine the gesture category of the matched gesture, determine and a control instruction corresponding to the gesture category, generating a virtual tool image corresponding to the control instruction, projecting a virtual image including the virtual tool image and the display image of the display device, and transmitting a control instruction to the display device, so that the display device executes the control instruction, thereby Implement the function corresponding to the control instruction.
  • the present disclosure can generate control commands by using a user's gestures, generate corresponding virtual tool images, and enable users to see images including virtual tools and display devices.
  • Display image virtual The image is simulated, and the control device can be used to control the display device to realize the corresponding function.
  • the human-computer interaction activity of the virtual reality technology can be realized without using the physical entity device to operate the display device, and the user who performs the virtual reality technology is improved. Operational convenience of machine interaction activities.
  • FIG. 1 is a schematic structural diagram of a human-machine interaction system according to Embodiment 1 of the present disclosure
  • FIG. 2 is a schematic structural diagram of a human-machine interaction system according to Embodiment 2 of the present disclosure
  • Embodiment 3 is a flowchart of a human-computer interaction method in Embodiment 3 of the present disclosure
  • Embodiment 4 is a flowchart of a human-computer interaction method in Embodiment 4 of the present disclosure
  • Embodiment 5 is a flowchart of a human-computer interaction method in Embodiment 5 of the present disclosure
  • Embodiment 6 is a visual effect diagram of a user in Embodiment 5 of the present disclosure.
  • FIG. 7 is a schematic structural diagram of a human-machine interaction apparatus according to Embodiment 6 of the present disclosure.
  • 10-human-computer interaction system 11-wearable device, 12-display device, 13-lens, 14-frame, 15-virtual pen, 16-smart glasses, 111-image acquisition module, 112-memory, 113-processor , 114 - image projection module, 115 - information transmission interface.
  • the human-machine interaction system 10 includes a wearable device 11 and a display device 12 .
  • the wearable device 11 includes an image acquisition module 111 , a memory 112 , a processor 113 , an image projection module 114 , and information transmission.
  • the image capturing module 111 is connected to the processor 113 for collecting the current gesture image of the user and the display image of the display device 12.
  • the image capturing module 111 can be an image capturing function such as a camera, a photographing lens or an image sensor.
  • the processor 113 is connected to the memory 112 for matching the current gesture image with the gesture in the gesture database pre-stored in the memory 112, determining the gesture category of the matched gesture, determining and The control command corresponding to the gesture category generates a virtual tool image corresponding to the control command according to the control command, and the virtual tool image is a virtual display image of a certain tool.
  • the memory 112 stores a gesture database
  • the gesture database includes In a specific implementation process, the gesture database also has a self-learning function, and the learned new gesture can be added to the gesture database; the image projection module 114 is connected to the processor 113.
  • the image projection module may project a current gesture image of the user in addition to the virtual image; the information sending interface 115 is connected to the processor 113.
  • the information sending interface 115 may be based on a communication technology used between the wearable device 11 and the display device 12, and the wearable device 11 and the display device 12 can be connected by wire, Can be connected via wireless, in the case of a wireless connection, For example, communication is performed by wireless communication means such as WiFi (Wireless-Fidelity), Bluetooth, ZigBee (Zigbee Protocol), etc., so that the corresponding information transmission interface 115 is used; the display device 12 is configured to perform control when receiving the control instruction.
  • the function of the control instruction is completed, and the function corresponding to the control instruction can be completed by the virtual tool.
  • the virtual tool image corresponding to the control instruction is an image of the virtual pen
  • the function corresponding to the control instruction implemented by the display device is virtual.
  • the wearable device 11 may be a device such as a smart eyeglass, a wearable helmet, or a smart wristband.
  • the display device 12 may be a portable display terminal such as a smart phone or a tablet computer, which is not limited thereto.
  • the image acquisition module 111 collects the current gesture image of the user and the display image of the display device 12, and the processor 113 matches the current gesture image with the gesture in the gesture database to determine the matching gesture.
  • the gesture category determines a control instruction corresponding to the gesture category, generates a virtual tool image corresponding to the control instruction, and the image projection module 114 projects a virtual image including the virtual tool image and the display image of the display device, and the information sending interface 115 transmits the display image to the display device 12
  • the control command is sent, and the display device 12 executes the control command, thereby implementing the function corresponding to the control command.
  • the present disclosure can generate control commands by using a user's gestures, generate corresponding virtual tool images, and enable users to see images including virtual tools and display devices. Displaying a virtual image of the image, and controlling the display device 12 to implement a corresponding function by using a control instruction, and realizing the human-computer interaction activity of the virtual reality technology without using the physical entity device to operate the display device 12, thereby improving the user's operation.
  • the operational convenience of human-computer interaction activities of virtual reality technology is compared with known solutions that require physical entity devices to implement human-computer interaction activities.
  • the smart glasses 16 include two lenses 13 corresponding to the left and right eyes and a lens frame 14 supporting the lenses. Both lenses 13 are near-eye lenses, and the display image on the display device 12 and the current gesture image of the user can be transmitted through the lens 13 to the wearer.
  • the eyes of the user of the smart glasses 16 can be integrated into the lens 13 to project a virtual image including the virtual tool image and the display image of the display device to the front of the user or directly on the lens 13 so that the user can pass
  • the smart glasses 16 view a virtual image including a virtual tool image and a display image of the display device; wherein the image capturing module 111, the memory 112, the processor 113, and the information transmitting interface 115 may all be disposed on the frame 14 of the smart glasses 16.
  • an embodiment of the present disclosure further provides a human-computer interaction method, which is applicable to the wearable device in the foregoing embodiment.
  • the wearable device may specifically be a smart glasses, a wearable helmet, or a smart bracelet.
  • human-computer interaction methods include:
  • step 201 the current gesture image of the user and the display image of the display device are collected. It should be noted that the combined image of the current gesture image and the display image of the display device is collected here, and then the display of the current gesture image and the display device is performed. The image is separated.
  • Step 202 Match the current gesture image with the gesture in the pre-stored gesture database, determine a gesture category of the matched gesture, and determine a control instruction corresponding to the gesture category.
  • the gesture database stores multiple gestures, and The current gesture image is matched with the gesture in the gesture database, and the gesture database is detected to have the same or similar gesture as the current gesture image, and the gesture category of the gesture that is the same or similar to the current gesture image is used as the gesture matching the current gesture image.
  • the gesture type may determine different control commands for different gesture categories. Specifically, the control instruction corresponding to the gesture category may be directly generated according to the gesture category, or the correspondence between the gesture category and the control instruction may be stored in advance, according to the current The gesture category invokes a control instruction corresponding to the current gesture category.
  • Step 203 Send a control instruction to the display device, so that the display device executes the control instruction; after receiving the control instruction sent by the wearable device, the display device executes the control instruction, thereby implementing a function corresponding to the control instruction.
  • Step 204 Generate a virtual tool image corresponding to the control instruction according to the control instruction.
  • the control instruction has a corresponding virtual tool
  • a virtual tool image that is, an image of the virtual tool, needs to be formed in the wearable device. , the virtual tools here are not actually there. The user sees the generated virtual tool image through the wearable device.
  • Step 205 projecting a virtual image including the virtual tool image and the display image of the display device; it should be noted that, in addition to the virtual image, the wearable device may also project the current gesture image of the user, so that the user can view through the wearable device.
  • the display image on the display device, and the current gesture image, and the image projected by the wearable device is based on the current gesture image captured by the wearable device and The display image of the display device is updated in real time, and the wearable device projects a coherent picture.
  • the current gesture image of the user and the display image of the display device are collected, the current gesture image is matched with the gesture in the gesture database, the gesture category of the matched gesture is determined, and the gesture category is determined.
  • Controlling an instruction generating a virtual tool image corresponding to the control instruction, projecting a virtual image including the virtual tool image and the display image of the display device, and transmitting a control instruction to the display device, so that the display device executes the control instruction, thereby implementing the control instruction corresponding The function.
  • the present disclosure can generate control commands by using a user's gestures, generate corresponding virtual tool images, and enable users to see images including virtual tools and display devices. Displaying a virtual image of the image, and controlling the display device to implement the corresponding function by using the control instruction, and realizing the human-computer interaction activity of the virtual reality technology without using the physical entity device to operate the display device, thereby improving the user's virtual reality The operational convenience of technical human-computer interaction activities.
  • the step 201 in the third embodiment can be specifically refined to step 2011 and step 2012.
  • the step 202 can be specifically refined to step 2020 to step 2024.
  • the specific content is as follows:
  • step 2011 the user's current gesture and the display image of the display device are photographed or photographed at the same time, and a mixed image including the current gesture image and the display image of the display device is acquired; if the user has not made a gesture, the wearable device has not been collected yet.
  • the wearable device first collects the display image of the display device, thereby obtaining the main environment in which the gesture made by the user in the current scene is located, and simultaneously collecting the current gesture of the user and the display device in the wearable device.
  • step 2012 is performed. It is worth mentioning that the specific way of acquiring the image may be taking a picture or taking a picture.
  • Step 2012 respectively identifying the current gesture image and the display image of the display device from the mixed image; specifically, the current gesture image and the display setting in the mixed image may be set by the background separation method
  • the current display image is separated to identify the current gesture image and the display image of the display device, respectively.
  • Step 2020 Obtain a gesture edge contour and a gesture motion trajectory of the current gesture image according to the current gesture image.
  • the gesture edge contour and the gesture motion trajectory are acquired from the current gesture image, and the gesture edge contour is used to represent the contour and the specific shape of the gesture.
  • the gesture motion track is used to indicate the direction of motion of the gesture and the changing process, etc., for example, the wearable device collects a gesture of the user's hand gripping into a solid fist, and moves linearly in front of the display device, wherein the contour of the gesture edge represents a solid fist
  • the contour of the hand is a shape similar to a circle, and the shape and position of the specific fingers of the solid fist.
  • the gesture movement track indicates that the solid fist is in a linear motion; for example, the wearable device collects the user's hand into a hollow fist
  • the contour of the edge of the gesture indicates that the outer contour of the hand gripped by the hollow fist is a shape similar to a circle, and the shape and position of the specific fingers of the hollow boxing
  • the data of the gesture movement indicates the hollow fist Curve movement; when distinguishing between solid fist and hollow fist, due to solid
  • the shape and position of each finger in the two gestures of boxing and hollow boxing are different, so the difference between the solid boxing and the hollow boxing can be distinguished by the contour of the edge of the gesture.
  • the features for characterizing the gesture include, but are not limited to, the gesture edge contour and the gesture motion trajectory described above, and different features capable of characterizing the gesture are all within the protection scope of the present disclosure.
  • Step 2021 Determine whether the gesture edge contour and the gesture motion track of the current gesture image match the gesture edge contour and the gesture motion track of the pre-stored gesture.
  • the gesture in the gesture database also has a gesture edge contour and a gesture motion track. And the like, comparing the gesture edge contour and the gesture motion trajectory of the current gesture image with the gesture edge contour of the pre-stored gesture and the gesture motion trajectory to determine whether the match is matched.
  • the gesture category of the pre-stored gesture is determined as the gesture category corresponding to the current gesture image; wherein the gesture category is an attribute of the gesture itself, used to represent the type of the gesture, for example, the gesture category, the marker gesture Categories, zoom gesture categories, and more.
  • step 2023 if there is no match, it is determined that the gesture category of the pre-stored gesture is not the gesture category corresponding to the current gesture image.
  • the step 202 in the third embodiment can be specifically refinement into steps 2025 to 2029 and step 2024.
  • the specific contents are as follows:
  • Step 2025 Acquire a gesture edge contour and a gesture motion trajectory of the current gesture image according to the current gesture image, and respectively establish a feature vector of the gesture edge contour and the gesture motion trajectory; wherein the gesture edge contour has a corresponding feature vector, and the gesture motion trajectory also has Corresponding feature vector.
  • Step 2026 extracting a feature value of the feature vector corresponding to the gesture edge contour and the gesture motion track; the wearable device performs segmentation and feature value extraction on the feature edge of the gesture edge contour and the gesture motion track, and extracts the gesture edge contour and the gesture motion track corresponding to the gesture edge contour The eigenvalue of the feature vector.
  • Step 2027 Determine whether the feature value corresponding to the gesture edge contour and the gesture motion track respectively matches the feature edge value of the pre-stored gesture and the feature value of the gesture motion track gesture; specifically, the gesture edge contour and the gesture motion of the gesture in the gesture database
  • the trajectory also has a feature vector, and the extracted feature values corresponding to the current gesture image can be matched with the feature values corresponding to the gestures in the gesture database, that is, the gesture edge contour of the current gesture image is determined and pre-stored. Whether the gesture edge contour of the gesture matches, and whether the gesture motion track of the current gesture image matches the gesture motion track of the pre-stored gesture.
  • Step 2028 if it matches, determining that the gesture category of the pre-stored gesture is the gesture category corresponding to the current gesture image; specifically, when the gesture edge contour of the current gesture image matches the gesture edge contour of the pre-stored gesture, and the gesture of the current gesture image The motion track is matched with the gesture motion track of the pre-stored gesture, and the gesture category of the pre-stored gesture is determined to be the gesture category corresponding to the current gesture image.
  • step 2029 if there is no match, it is determined that the gesture category of the pre-stored gesture is not the gesture category corresponding to the current gesture image; when the gesture edge contour of the current gesture image does not match the gesture edge contour of the pre-stored gesture, or the gesture motion of the current gesture image When the trajectory does not match the gesture trajectory of the pre-stored gesture, or both of the above does not match, the gesture category of the pre-stored gesture is determined not to be the gesture category corresponding to the current gesture image.
  • Step 2024 Search and obtain a control instruction corresponding to the gesture category according to the correspondence between the pre-stored gesture category and the control instruction. For details, refer to Embodiment 4, and details are not described herein again.
  • the smart glasses 16 use the camera to collect images
  • the display device 12 is a tablet computer
  • the camera on the smart glasses 16 collects the text reading interface displayed on the tablet computer, and obtains the display image of the tablet computer; the user wants to mark a certain sentence text in the text reading interface, so the user makes a grip. Pen gesture, want to be in The position corresponding to the sentence text is marked. It should be noted that there is no real physical tool such as a touch pen in the user's hand.
  • the camera collects the text reading interface displayed by the tablet computer and the user's pen gesture, thereby obtaining the display image.
  • the smart glasses 16 respectively establish a feature vector for the gesture edge contour and the gesture motion track of the pen gesture, and extract the feature values from the feature vector, and perform matching in the gesture database stored in the smart glasses 16, thereby
  • the gesture category of the gesture database that obtains the gesture matching the extracted feature value is a pen-marking gesture, and the control command corresponding to the gesture of the pen-marking gesture is transmitted to the tablet computer, and the tablet computer controls the gesture according to the pen-marking gesture.
  • the image of the virtual pen 15 is formed in the field of view of the smart glasses 16 and is marked at the text corresponding to the pen gesture. It should be noted that the smart glasses 16 generate an image of the virtual pen 15 so that the user can see the virtual pen through the smart glasses 16 . 15 images, the effect picture seen by the user is Figure 6, with the user The same effect as a real physical touch pen tool marks.
  • the gesture type of the gesture is a zoom-in gesture, and the control instruction corresponding to the zoom gesture is transmitted to the tablet, and the tablet forms an image of the virtual magnifying glass in the field of view of the smart glasses according to the control instruction of the zoom gesture, and the text reading interface or The game interface is enlarged and used. You can see a virtual magnifying glass to enlarge the image and text reading interface or game interface through smart glasses.
  • the smart glasses use the gesture edge contour and the gesture motion track to identify and match the collected gestures, generate a control instruction according to the target gesture, generate a virtual tool image according to the control instruction, and display device execution control.
  • the functions corresponding to the instructions make the human-computer interaction activities more precise and complete.
  • the collecting module 31 is configured to collect a current gesture image of the user and a display image of the display device;
  • a determining module 32 configured to enter the current gesture image with a gesture in a pre-stored gesture database Matching, determining a gesture category of the matched gesture, and determining a control instruction corresponding to the gesture category;
  • a sending module 33 configured to send a control instruction to the display device, so that the display device executes the control instruction
  • a generating module 34 configured to generate a virtual tool image corresponding to the control instruction according to the control instruction
  • the projection module 35 is configured to project a virtual image including the virtual tool image and a display image of the display device.
  • the acquisition module 31 collects the current gesture image of the user and the display image of the display device, and the determination module 32 matches the current gesture image with the gesture in the gesture database to determine the gesture category of the matching gesture. Determining a control instruction corresponding to the gesture category, the generating module 34 generates a virtual tool image corresponding to the control instruction, the projection module 35 projects a virtual image including the virtual tool image and the display image of the display device, and the sending module 33 sends a control instruction to the display device. And causing the display device to execute the control instruction, thereby implementing a function corresponding to the control instruction.
  • the present disclosure can generate control commands by using a user's gestures, generate corresponding virtual tool images, and enable users to see images including virtual tools and display devices. Displaying a virtual image of the image, and controlling the display device to implement the corresponding function by using the control instruction, and realizing the human-computer interaction activity of the virtual reality technology without using the physical entity device to operate the display device, thereby improving the user's virtual reality The operational convenience of technical human-computer interaction activities.
  • the acquiring module 31 is specifically configured to simultaneously take a picture or a camera to the current gesture of the user and the display image of the display device, acquire a mixed image including the current gesture image and the display image of the display device, and respectively identify the current gesture image and the display from the mixed image.
  • the display image of the device is specifically configured to simultaneously take a picture or a camera to the current gesture of the user and the display image of the display device, acquire a mixed image including the current gesture image and the display image of the display device, and respectively identify the current gesture image and the display from the mixed image.
  • the determining module 32 is specifically configured to find and obtain a control instruction corresponding to the gesture category according to the pre-stored correspondence between the gesture category and the control instruction.
  • the determining module 32 is further configured to: acquire a gesture edge contour and a gesture motion track of the current gesture image according to the current gesture image; determine a gesture edge contour and a gesture motion track of the current gesture image, and a gesture edge contour and a gesture of the pre-stored gesture Whether the motion trajectory matches; when matching, the gesture category of the pre-stored gesture is determined as the gesture category corresponding to the current gesture image; when not matched, the gesture category of the pre-stored gesture is determined not to be the gesture category corresponding to the current gesture image.
  • the determining module 32 is further configured to: acquire the current gesture image according to the current gesture image.
  • Gesture edge contour and gesture motion trajectory respectively establish feature vector of gesture edge contour and gesture motion trajectory; extract feature value of feature edge corresponding to gesture edge contour and gesture motion trajectory; determine whether feature value corresponding to gesture edge contour and gesture motion trajectory is And respectively matching the feature edge value of the pre-stored gesture with the feature value of the gesture motion track gesture; when matching, determining the gesture category of the pre-existing gesture as the gesture category corresponding to the current gesture image; when not matching, determining the gesture of the pre-existing gesture The category is not the gesture category corresponding to the current gesture image.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Hardware Design (AREA)
  • Optics & Photonics (AREA)
  • Multimedia (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

一种人机交互系统、方法及装置，涉及人机交互技术领域。所述人机交互系统（10），包括可穿戴设备（11）和显示设备（12）；可穿戴设备（11）包括：图像采集模块（111）、存储器（112）、处理器（113）、图像投射模块（114）、信息发送接口（115），图像采集模块（111）用于采集用户的当前手势图像和显示设备的显示图像；处理器（113）用于将当前手势图像与存储器（112）中预存的手势数据库中的手势进行匹配，确定匹配的手势的手势类别，确定与手势类别对应的控制指令，根据控制指令，生成与控制指令对应的虚拟工具图像；图像投射模块（114）用于投射出包括虚拟工具图像和显示图像的虚拟图像；信息发送接口（115）用于将所述控制指令发送给所述显示设备（12）。

Description

人机交互系统、方法及装置 技术领域
本公开涉及人机交互技术领域，尤其涉及一种人机交互系统、方法及装置。
背景技术
随着电子信息科学技术的发展，人与计算机的交互活动在人们的日常生活中占据了重要的地位，其中，结合虚拟现实技术利用人的动作进行人机交互活动也成为了人机交互活动的一个主要发展趋势。
如已知的，结合虚拟现实技术进行人机交互活动时，用户需要与计算机直接接触，或者使用实体输入装置对计算机发出操作指令。比如，用户对计算机进行触摸操作，从而实现计算机的各种功能，或者，计算机连接有鼠标和键盘，用户利用鼠标和键盘对计算机发出操作指令，从而实现计算机的各种功能。但是，由于用户需要使用实体物理装置来操作计算机的方式较为复杂，使得用户进行虚拟现实技术的人机交互活动的操作便利性较低。
发明内容
本公开提供了一种人机交互系统、方法及装置，用于提高用户进行虚拟现实技术的人机交互活动的操作便利性。
相应地，本公开提供了如下技术方案：
第一方面，本公开提供了一种人机交互系统，包括：可穿戴设备和显示设备；
所述可穿戴设备包括图像采集模块、存储器、处理器、图像投射模块、信息发送接口；
所述图像采集模块连接所述处理器，用于采集用户的当前手势图像和所述显示设备的显示图像；
所述处理器连接所述存储器，用于将所述当前手势图像与所述存储器中预存的手势数据库中的手势进行匹配，确定匹配的手势的手势类别，确定与所述手势类别对应的控制指令，根据所述控制指令，生成与所述控制指令对应的虚拟工具图像；
所述图像投射模块连接所述处理器，用于投射出包括所述虚拟工具图像和所述显示图像的虚拟图像；
所述信息发送接口连接所述处理器，用于将所述控制指令发送给所述显示设备；
所述显示设备用于在接收到所述控制指令时，执行所述控制指令。
第二方面，本公开提供了一种人机交互方法，包括：
可穿戴设备采集用户的当前手势图像和显示设备的显示图像；
将所述当前手势图像与预存的手势数据库中的手势进行匹配，确定匹配的手势的手势类别，确定与所述手势类别对应的控制指令；
根据所述控制指令，生成与所述控制指令对应的虚拟工具图像，投射出包括所述虚拟工具图像和所述显示设备的显示图像的虚拟图像；
将所述控制指令发送给所述显示设备，使得所述显示设备执行所述控制指令。
第三方面，本公开提供了一种人机交互装置，包括：
采集模块，用于采集用户的当前手势图像和显示设备的显示图像；
确定模块，用于将所述当前手势图像与预存的手势数据库中的手势进行匹配，确定匹配的手势的手势类别，确定与所述手势类别对应的控制指令；
生成模块，用于根据所述控制指令，生成与所述控制指令对应的虚拟工具图像；
投射模块，用于投射出包括所述虚拟工具图像和所述显示设备的显示图像的虚拟图像；
发送模块，用于将所述控制指令发送给所述显示设备，使得所述显示设备执行所述控制指令。
本公开提供的人机交互系统、方法及装置中，采集用户的当前手势图像和显示设备的显示图像，将当前手势图像与手势数据库中的手势进行匹配，确定匹配的手势的手势类别，确定与手势类别对应的控制指令，生成与控制指令对应的虚拟工具图像，投射出包括虚拟工具图像和显示设备的显示图像的虚拟图像，并向显示设备发送控制指令，使得显示设备执行该控制指令，从而实现控制指令对应的功能。与需要利用物理实体装置实现人机交互活动的已知方案相比，本公开利用用户的手势就能生成控制指令，生成对应的虚拟工具图像，使用户能够看到包括虚拟工具图像和显示设备的显示图像的虚 拟图像，并能利用控制指令控制显示设备实现对应的功能，不需要特意使用物理实体装置对显示设备进行操作，就能够实现虚拟现实技术的人机交互活动，提高了用户进行虚拟现实技术的人机交互活动的操作便利性。
附图说明
此处所说明的附图用来提供对本公开的进一步理解，构成本公开的一部分，本公开的示意性实施例及其说明用于解释本公开，并不构成对本公开的不当限定。在附图中：
图1为本公开实施例一中人机交互系统的结构示意图；
图2为本公开实施例二中人机交互系统的结构示意图；
图3为本公开实施例三中人机交互方法的流程图；
图4为本公开实施例四中人机交互方法的流程图；
图5为本公开实施例五中人机交互方法的流程图；
图6为本公开实施例五中用户的视觉效果图；
图7为本公开实施例六中人机交互装置的结构示意图。
附图标记：
10-人机交互系统，11-可穿戴设备，12-显示设备，13-镜片，14-镜框，15-虚拟笔，16-智能眼镜，111-图像采集模块，112-存储器，113-处理器，114-图像投射模块，115-信息发送接口。
具体实施方式
为了进一步说明本公开实施例提供的人机交互系统及方法，下面结合说明书附图进行详细描述。
实施例一
请参阅图1，本公开实施例提供的人机交互系统10包括可穿戴设备11和显示设备12，可穿戴设备11包括图像采集模块111、存储器112、处理器113、图像投射模块114、信息发送接口115；图像采集模块111连接处理器113，用于采集用户的当前手势图像和显示设备12的显示图像，具体的，图像采集模块111可以为摄像头、拍照镜头或图像传感器等具有图像采集功能的部件；处理器113连接存储器112，用于将当前手势图像与存储器112中预存的手势数据库中的手势进行匹配，确定匹配的手势的手势类别，确定与 手势类别对应的控制指令，根据控制指令，生成与控制指令对应的虚拟工具图像，虚拟工具图像为某种工具的虚拟显示图像，需要说明的是，存储器112中存储有手势数据库，手势数据库中包括多种手势以及多种手势各自的相关数据，在具体实施过程中，手势数据库也具有自学习的功能，能够将学习到的新的手势添加到手势数据库中；图像投射模块114连接处理器113，用于投射出包括所述虚拟工具图像和所述显示图像的虚拟图像，根据具体需求，除了上述虚拟图像，图像投射模块还可以投射出用户的当前手势图像；信息发送接口115连接处理器113，用于将控制指令发送给显示设备12，信息发送接口115可以根据可穿戴设备11与显示设备12之间使用的通信技术而定，可穿戴设备11与显示设备12之间能够通过有线连接，也可以通过无线连接，在无线连接的情况下，比如利用WiFi(Wireless-Fidelity，无线宽带)、蓝牙、ZigBee(紫蜂协议)等无线通讯手段进行通信，从而使用对应的信息发送接口115；显示设备12用于在接收到控制指令时，执行控制指令，从而完成控制指令对应的功能，控制指令对应的功能能够由虚拟工具完成，比如，与控制指令对应的虚拟工具图像为虚拟笔的图像，则显示装置实现的与控制指令对应的功能为虚拟笔的书写、标记功能。具体的，可穿戴设备11具体可以为智能眼镜、穿戴头盔、智能手环等设备，显示设备12具体可以为智能手机、平板电脑等便于携带的手持显示终端，在此并不限定。
本公开提供的人机交互系统10中，图像采集模块111采集用户的当前手势图像和显示设备12的显示图像，处理器113将当前手势图像与手势数据库中的手势进行匹配，确定匹配的手势的手势类别，确定与手势类别对应的控制指令，生成与控制指令对应的虚拟工具图像，图像投射模块114投射出包括虚拟工具图像和显示设备的显示图像的虚拟图像，信息发送接口115向显示设备12发送控制指令，显示设备12执行该控制指令，从而实现控制指令对应的功能。与需要利用物理实体装置实现人机交互活动的已知方案相比，本公开利用用户的手势就能生成控制指令，生成对应的虚拟工具图像，使用户能够看到包括虚拟工具图像和显示设备的显示图像的虚拟图像，并能利用控制指令控制显示设备12实现对应的功能，不需要特意使用物理实体装置对显示设备12进行操作，就能够实现虚拟现实技术的人机交互活动，提高了用户进行虚拟现实技术的人机交互活动的操作便利性。
实施例二
在实施例一的基础上，请参阅图2，以可穿戴设备11为智能眼镜16为例来进行说明。智能眼镜16包括与左右眼对应的两个镜片13以及支持镜片的镜框14，两个镜片13均为近眼透镜，显示设备12上的显示图像以及用户的当前手势图像均可以通过镜片13透射至佩戴智能眼镜16的用户的双眼，图像投射模块114可以集成于镜片13中，将包括虚拟工具图像和显示设备的显示图像的虚拟图像投影至用户眼前，或直接显示在镜片13上，使得用户能够通过智能眼镜16观看到包括虚拟工具图像和显示设备的显示图像的虚拟图像；其中，图像采集模块111、存储器112、处理器113和信息发送接口115均可设置在智能眼镜16的镜框14上。
实施例三
请参阅图3，本公开实施例还提供了一种人机交互方法，该人机交互方法应用于上述实施例中的可穿戴设备，可穿戴设备具体可以为智能眼镜、穿戴头盔、智能手环等设备，人机交互方法包括：
步骤201，采集用户的当前手势图像和显示设备的显示图像；需要说明的是，这里采集到的是当前手势图像和显示设备的显示图像的组合图像，之后再对当前手势图像和显示设备的显示图像进行分离。
步骤202，将当前手势图像与预存的手势数据库中的手势进行匹配，确定匹配的手势的手势类别，确定与手势类别对应的控制指令；需要说明的是，手势数据库中存储有多种手势，将当前手势图像与手势数据库中的手势进行匹配，检测手势数据库中是否存在与当前手势图像相同或相似的手势，将与当前手势图像相同或相似的手势的手势类别，作为与当前手势图像匹配的手势的手势类别，针对不同的手势类别可以确定不同的控制指令，具体的，可以根据手势类别，直接生成与该手势类别对应的控制指令，也可以预先存储手势类别与控制指令的对应关系，根据当前的手势类别，调用与当前的手势类别对应的控制指令。
步骤203，将控制指令发送给显示设备，使得显示设备执行控制指令；显示设备在接收到可穿戴设备发送的控制指令后，执行控制指令，从而实现与控制指令对应的功能。
步骤204，根据控制指令，生成与控制指令对应的虚拟工具图像；需要说明的是，控制指令具有对应的虚拟工具，在步骤204中需要在可穿戴设备中形成虚拟工具图像，即虚拟工具的图像，这里的虚拟工具并不是实际存在 的，用户是通过可穿戴设备看到生成的虚拟工具图像的。
步骤205，投射出包括虚拟工具图像和显示设备的显示图像的虚拟图像；需要说明的是，除了上述虚拟图像外，可穿戴设备还可以投射用户的当前手势图像，使得用户能够通过可穿戴设备观看到虚拟工具图像和显示设备上的显示图像，或者观看到虚拟工具图像、显示设备上的显示图像和当前手势图像，且可穿戴设备投射出的图像会根据可穿戴设备采集到的当前手势图像和显示设备的显示图像进行实时更新，可穿戴设备会投射出连贯的画面。
本公开提供的人机交互方法中，采集用户的当前手势图像和显示设备的显示图像，将当前手势图像与手势数据库中的手势进行匹配，确定匹配的手势的手势类别，确定与手势类别对应的控制指令，生成与控制指令对应的虚拟工具图像，投射出包括虚拟工具图像和显示设备的显示图像的虚拟图像，并向显示设备发送控制指令，使得显示设备执行该控制指令，从而实现控制指令对应的功能。与需要利用物理实体装置实现人机交互活动的已知方案相比，本公开利用用户的手势就能生成控制指令，生成对应的虚拟工具图像，使用户能够看到包括虚拟工具图像和显示设备的显示图像的虚拟图像，并能利用控制指令控制显示设备实现对应的功能，不需要特意使用物理实体装置对显示设备进行操作，就能够实现虚拟现实技术的人机交互活动，提高了用户进行虚拟现实技术的人机交互活动的操作便利性。
实施例四
请参阅图4，在实施例三的基础上，实施例三中的步骤201能够具体细化为步骤2011和步骤2012，步骤202能够具体细化为步骤2020-步骤2024，具体内容如下：
步骤2011，同时对用户的当前手势和显示设备的显示图像进行拍照或摄像，获取包含当前手势图像和显示设备的显示图像的混合图像；在用户还未做出手势，即可穿戴设备还未采集到用户的当前手势图像时，可穿戴设备先采集显示设备的显示图像，从而得到当前场景下用户做出的手势所处的主体环境，在可穿戴设备同时采集到用户的当前手势和显示设备的显示图像的混合图像时，执行步骤2012，值得一提的是，采集图像的具体方式可以为拍照或摄像。
步骤2012，从混合图像中分别识别出当前手势图像和显示设备的显示图像；具体的，可以通过背景分离方法将混合图像中的当前手势图像和显示设 备当前的显示图像分离开来，从而分别识别出当前手势图像和显示设备的显示图像。
步骤2020，根据当前手势图像，获取当前手势图像的手势边缘轮廓和手势运动轨迹；其中，从当前手势图像中可以获取手势边缘轮廓和手势运动轨迹，手势边缘轮廓用于表示手势的轮廓和具体形状，手势运动轨迹用于表示手势的运动方向以及变化过程等，比如：可穿戴设备采集到用户的手握成实心拳，在显示设备前直线移动的手势，其中，手势边缘轮廓表示握成实心拳的手的轮廓为近似于圆形的形状，以及实心拳的具体各个手指的形状和位置，手势运动轨迹表示实心拳呈直线运动；再比如，可穿戴设备采集到用户的手握成空心拳，在显示设备前曲线移动的手势，手势边缘轮廓表示握成空心拳的手的外部轮廓为近似于圆形的形状，以及空心拳的具体各个手指的形状和位置，手势运动轨迹数据表示空心拳呈曲线运动；当区分实心拳与空心拳时，由于实心拳与空心拳这两个手势中各个手指的形状和位置不同，故可以由手势边缘轮廓来区分实心拳和空心拳的不同。值得一提的是，用于表征手势的特征包括但并不限于上述手势边缘轮廓和手势运动轨迹，能够表征手势的不同的特征均属于本公开的保护范围。
步骤2021，判断当前手势图像的手势边缘轮廓和手势运动轨迹，与预存的手势的手势边缘轮廓和手势运动轨迹是否匹配；需要说明的是，手势数据库中的手势也具有手势边缘轮廓和手势运动轨迹等特征，将当前手势图像的手势边缘轮廓和手势运动轨迹，与预存的手势的手势边缘轮廓和手势运动轨迹进行对比，判断是否匹配。
步骤2022，若匹配，则确定预存的手势的手势类别为当前手势图像对应的手势类别；其中，手势类别是手势自身的一种属性，用于表征手势的种类，比如，开启手势类别、标记手势类别、放大手势类别等等。
步骤2023，若不匹配，则确定预存的手势的手势类别不是当前手势图像对应的手势类别。
步骤2024，根据预存的所手势类别和控制指令的对应关系，查找并得到与手势类别对应的控制指令；需要说明的是，可穿戴设备中还可以预存有手势类别和控制指令的对应关系，当确定手势类别后，查找到与手势类别对应的控制指令，便于进行调用。
实施例五
请参阅图5，在实施例三和实施例四的基础上，实施例三中的步骤202能够具体细化为步骤2025-步骤2029和步骤2024，具体内容如下：
步骤2025，根据当前手势图像，获取当前手势图像的手势边缘轮廓和手势运动轨迹，分别建立手势边缘轮廓和手势运动轨迹的特征向量；其中，手势边缘轮廓具有对应的特征向量，手势运动轨迹也具有对应的特征向量。
步骤2026，提取手势边缘轮廓和手势运动轨迹对应的特征向量的特征值；可穿戴设备对手势边缘轮廓和手势运动轨迹的特征向量进行分割与特征值提取，提取出手势边缘轮廓和手势运动轨迹对应的特征向量的特征值。
步骤2027，判断手势边缘轮廓和手势运动轨迹对应的特征值是否分别与预存的手势的手势边缘轮廓和手势运动轨迹手势的特征值匹配；具体的，手势数据库中的手势的手势边缘轮廓和手势运动轨迹也具有特征向量，可以利用提取出的与当前手势图像对应的各特征值，与手势数据库中的手势对应的各特征值进行匹配，也就是说，判断当前手势图像的手势边缘轮廓与预存的手势的手势边缘轮廓是否匹配，以及当前手势图像的手势运动轨迹与预存的手势的手势运动轨迹是否匹配。
步骤2028，若匹配，则确定预存的手势的手势类别为当前手势图像对应的手势类别；具体的，当当前手势图像的手势边缘轮廓与预存的手势的手势边缘轮廓匹配，且当前手势图像的手势运动轨迹与预存的手势的手势运动轨迹匹配，即可确定预存的手势的手势类别为当前手势图像对应的手势类别。
步骤2029，若不匹配，则确定预存的手势的手势类别不是当前手势图像对应的手势类别；当当前手势图像的手势边缘轮廓与预存的手势的手势边缘轮廓不匹配，或者当前手势图像的手势运动轨迹与预存的手势的手势运动轨迹不匹配，或者上述两者均不匹配时，确定预存的手势的手势类别不是当前手势图像对应的手势类别。
步骤2024，根据预存的手势类别和控制指令的对应关系，查找并得到与手势类别对应的控制指令；具体内容参见实施例四，在此不再赘述。
下面以一具体举例来进行本实施例中人机交互系统实现人机交互活动的说明：如图2所示，智能眼镜16利用摄像头采集图像，显示设备12为平板电脑，平板电脑上显示的画面是文字阅读界面，智能眼镜16上的摄像头对平板电脑显示的文字阅读界面进行采集，得到平板电脑的显示图像；用户此时希望对文字阅读界面中的某句文字进行标记，故用户做出握笔手势，想要在 该句文字对应的位置做出标记，需要注意的是，用户手中不存在触摸笔等真实物理工具，此时，摄像头采集到了平板电脑显示的文字阅读界面和用户的握笔手势，从而得到显示图像和当前手势图像；智能眼镜16分别对握笔手势的手势边缘轮廓和手势运动轨迹建立特征向量，并从上述特征向量中提取特征值，在智能眼镜16中存储的手势数据库中进行匹配，从而在手势数据库中得到与提取的特征值匹配的手势的手势类别为握笔标记手势，并将手势为握笔标记手势对应的控制指令传送给平板电脑，平板电脑根据握笔标记手势的控制指令，在智能眼镜16的视野范围内形成虚拟笔15的图像，在握笔手势对应的文字处进行标记，需要注意的是，智能眼镜16生成虚拟笔15的图像，使得用户可以通过智能眼镜16看到虚拟笔15的图像，用户看到的效果图为图6，与用户使用触摸笔等真实物理工具进行标记的效果相同。
下面再以另一具体举例来进行本实施例中人机交互系统实现人机交互活动的说明：智能眼镜利用摄像头采集图像，显示设备为平板电脑，平板电脑上显示的画面是文字阅读界面或游戏界面，用户需要将文字阅读界面或游戏界面放大，则可做出手持放大镜的手势或加号手势，摄像头采集到手持放大镜的手势图像或加号手势图像后，智能眼镜分别对放大镜的手势图像或加号手势图像的手势边缘轮廓和手势运动轨迹建立特征向量，并从上述特征向量中提取特征值，在智能眼镜中存储的手势数据库中进行匹配，从而得到在手势数据库中与提取的特征值匹配的手势的手势类别为放大手势，并将放大手势对应的控制指令传送给平板电脑，平板电脑根据放大手势的控制指令，在智能眼镜的视野范围内形成虚拟放大镜的图像，并将文字阅读界面或游戏界面放大，用户可以通过智能眼镜看到虚拟放大镜的图像以及放大的文字阅读界面或游戏界面。
本实施例中的人机交互系统，智能眼镜利用手势边缘轮廓和手势运动轨迹对采集到的手势进行识别和匹配，并根据目标手势生成控制指令，根据控制指令生成虚拟工具图像，显示设备执行控制指令对应的功能，使得人机交互活动更加精准、完善。
实施例六
请参照图7，本公开实施例还提供了一种人机交互装置30，包括：
采集模块31，用于采集用户的当前手势图像和显示设备的显示图像；
确定模块32，用于将所述当前手势图像与预存的手势数据库中的手势进 行匹配，确定匹配的手势的手势类别，确定与所述手势类别对应的控制指令；
发送模块33，用于将控制指令发送给显示设备，使得显示设备执行控制指令；
生成模块34，用于根据控制指令，生成与控制指令对应的虚拟工具图像；
投射模块35，用于投射出包括所述虚拟工具图像和所述显示设备的显示图像的虚拟图像。
本公开提供的人机交互装置30中，采集模块31采集用户的当前手势图像和显示设备的显示图像，确定模块32将当前手势图像与手势数据库中的手势进行匹配，确定匹配手势的手势类别，确定与手势类别对应的控制指令，生成模块34生成与控制指令对应的虚拟工具图像，投射模块35投射出包括虚拟工具图像和显示设备的显示图像的虚拟图像，发送模块33向显示设备发送控制指令，使得显示设备执行该控制指令，从而实现控制指令对应的功能。与需要利用物理实体装置实现人机交互活动的已知方案相比，本公开利用用户的手势就能生成控制指令，生成对应的虚拟工具图像，使用户能够看到包括虚拟工具图像和显示设备的显示图像的虚拟图像，并能利用控制指令控制显示设备实现对应的功能，不需要特意使用物理实体装置对显示设备进行操作，就能够实现虚拟现实技术的人机交互活动，提高了用户进行虚拟现实技术的人机交互活动的操作便利性。
实施例七
在实施例六的基础上，下面将介绍上述各个模块的具体操作：
采集模块31具体用于同时对用户的当前手势和显示设备的显示图像进行拍照或摄像，获取包含当前手势图像和显示设备的显示图像的混合图像；从混合图像中分别识别出当前手势图像和显示设备的显示图像。
确定模块32具体用于根据预存的所述手势类别和控制指令的对应关系，查找并得到与所述手势类别对应的控制指令。
确定模块32还可以具体用于：根据当前手势图像，获取当前手势图像的手势边缘轮廓和手势运动轨迹；判断当前手势图像的手势边缘轮廓和手势运动轨迹，与预存的手势的手势边缘轮廓和手势运动轨迹是否匹配；当匹配时，确定预存的手势的手势类别为当前手势图像对应的手势类别；当不匹配时，确定预存的手势的手势类别不是当前手势图像对应的手势类别。
确定模块32还可以具体用于：根据当前手势图像，获取当前手势图像的 手势边缘轮廓和手势运动轨迹，分别建立手势边缘轮廓和手势运动轨迹的特征向量；提取手势边缘轮廓和手势运动轨迹对应的特征向量的特征值；判断手势边缘轮廓和手势运动轨迹对应的特征值是否分别与预存的手势的手势边缘轮廓和手势运动轨迹手势的特征值匹配；当匹配时，确定预存的手势的手势类别为当前手势图像对应的手势类别；当不匹配时，确定预存的手势的手势类别不是当前手势图像对应的手势类别。
本说明书中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。对于人机交互装置的实施例而言，由于其基本相似于人机交互方法实施例，所以描述得比较简单，相关之处参见人机交互方法实施例的部分说明即可。
在上述实施方式的描述中，具体特征、结构、材料或者特点可以在任何的一个或多个实施例或示例中以合适的方式结合。
以上所述，仅为本公开的具体实施方式，但本公开的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本公开揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本公开的保护范围之内。因此，本公开的保护范围应以所述权利要求的保护范围为准。
本申请要求于2016年1月4日递交的中国专利申请第201610004024.3号的优先权，在此全文引用上述中国专利申请公开的内容以作为本申请的一部分。

Claims (12)

  1. 一种人机交互系统，包括：可穿戴设备和显示设备；
    所述可穿戴设备包括图像采集模块、存储器、处理器、图像投射模块、信息发送接口；
    所述图像采集模块连接所述处理器，用于采集用户的当前手势图像和所述显示设备的显示图像；
    所述处理器连接所述存储器，用于将所述当前手势图像与所述存储器中预存的手势数据库中的手势进行匹配，确定匹配的手势的手势类别，确定与所述手势类别对应的控制指令，根据所述控制指令，生成与所述控制指令对应的虚拟工具图像；
    所述图像投射模块连接所述处理器，用于投射出包括所述虚拟工具图像和所述显示图像的虚拟图像；
    所述信息发送接口连接所述处理器，用于将所述控制指令发送给所述显示设备；
    所述显示设备用于在接收到所述控制指令时，执行所述控制指令。
  2. 根据权利要求1所述的人机交互系统，其中，所述可穿戴设备为智能眼镜，所述智能眼镜包括镜片和支撑所述镜片的镜框，所述图像投射模块集成在所述镜片中，所述图像采集模块设置在所述镜框上。
  3. 一种人机交互方法，包括：
    可穿戴设备采集用户的当前手势图像和显示设备的显示图像；
    将所述当前手势图像与预存的手势数据库中的手势进行匹配，确定匹配的手势的手势类别，确定与所述手势类别对应的控制指令；
    根据所述控制指令，生成与所述控制指令对应的虚拟工具图像；
    投射出包括所述虚拟工具图像和所述显示设备的显示图像的虚拟图像；
    将所述控制指令发送给所述显示设备，使得所述显示设备执行所述控制指令。
  4. 根据权利要求3所述的人机交互方法，其中，采集用户的当前手势图像和显示设备的显示图像，包括：
    同时对用户的当前手势和所述显示设备的显示图像进行拍照或摄像，获 取包含所述当前手势图像和所述显示设备的显示图像的混合图像；
    从所述混合图像中分别识别出所述当前手势图像和所述显示设备的显示图像。
  5. 根据权利要求3或4所述的人机交互方法，其中，确定与所述手势类别对应的控制指令，包括：
    根据预存的所述手势类别和控制指令的对应关系，查找并得到与所述手势类别对应的控制指令。
  6. 根据权利要求3或4所述的人机交互方法，其中，将所述当前手势图像与预存的手势数据库中的手势进行匹配，确定匹配的手势的手势类别，包括：
    根据所述当前手势图像，获取所述当前手势图像的手势边缘轮廓和手势运动轨迹；
    判断所述当前手势图像的手势边缘轮廓和手势运动轨迹，与预存的手势的手势边缘轮廓和手势运动轨迹是否匹配；
    若匹配，则确定所述预存的手势的手势类别为所述当前手势图像对应的手势类别；
    若不匹配，则确定所述预存的手势的手势类别不是所述当前手势图像对应的手势类别。
  7. 根据权利要求3或4所述的人机交互方法，其中，将所述当前手势图像与预存的手势数据库中的手势进行匹配，确定匹配的手势的手势类别，包括：
    根据所述当前手势图像，获取所述当前手势图像的手势边缘轮廓和手势运动轨迹，分别建立所述手势边缘轮廓和所述手势运动轨迹的特征向量；
    提取所述手势边缘轮廓和所述手势运动轨迹对应的特征向量的特征值；
    判断所述手势边缘轮廓和所述手势运动轨迹对应的特征值是否分别与预存的手势的手势边缘轮廓和手势运动轨迹手势的特征值匹配；
    若匹配，确定所述预存的手势的手势类别为所述当前手势图像对应的手势类别；
    若不匹配，确定所述预存的手势的手势类别不是所述当前手势图像对应的手势类别。
  8. 一种人机交互装置，包括：
    采集模块，用于采集用户的当前手势图像和显示设备的显示图像；
    确定模块，用于将所述当前手势图像与预存的手势数据库中的手势进行匹配，确定匹配的手势的手势类别，确定与所述手势类别对应的控制指令；
    生成模块，用于根据所述控制指令，生成与所述控制指令对应的虚拟工具图像；
    投射模块，用于投射出包括所述虚拟工具图像和所述显示设备的显示图像的虚拟图像；
    发送模块，用于将所述控制指令发送给所述显示设备，使得所述显示设备执行所述控制指令。
  9. 根据权利要求8所述的人机交互装置，其中，所述采集模块用于：同时对用户的当前手势和所述显示设备的显示图像进行拍照或摄像，获取包含所述当前手势图像和所述显示设备的显示图像的混合图像；从所述混合图像中分别识别出所述当前手势图像和所述显示设备的显示图像。
  10. 根据权利要求8或9所述的人机交互装置，其中，所述确定模块用于：根据预存的所述手势类别和控制指令的对应关系，查找并得到与所述手势类别对应的控制指令。
  11. 根据权利要求8或9所述的人机交互装置，其中，所述确定模块用于：根据所述当前手势图像，获取所述当前手势图像的手势边缘轮廓和手势运动轨迹；判断所述当前手势图像的手势边缘轮廓和手势运动轨迹，与预存的手势的手势边缘轮廓和手势运动轨迹是否匹配；当匹配时，确定所述预存的手势的手势类别为所述当前手势图像对应的手势类别；当不匹配时，确定所述预存的手势的手势类别不是所述当前手势图像对应的手势类别。
  12. 根据权利要求8或9所述的人机交互装置，其中，所述确定模块用于：根据所述当前手势图像，获取所述当前手势图像的手势边缘轮廓和手势运动轨迹，分别建立所述手势边缘轮廓和所述手势运动轨迹的特征向量；提取所述手势边缘轮廓和所述手势运动轨迹对应的特征向量的特征值；判断所述手势边缘轮廓和所述手势运动轨迹对应的特征值是否分别与预存的手势的手势边缘轮廓和手势运动轨迹手势的特征值匹配；当匹配时，确定所述预存的手势的手势类别为所述当前手势图像对应的手势类别；当不匹配时，确定所述预存的手势的手势类别不是所述当前手势图像对应的手势类别。
PCT/CN2016/098568 2016-01-04 2016-09-09 人机交互系统、方法及装置 WO2017118075A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/519,689 US10585488B2 (en) 2016-01-04 2016-09-09 System, method, and apparatus for man-machine interaction

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610004024.3 2016-01-04
CN201610004024.3A CN105487673B (zh) 2016-01-04 2016-01-04 一种人机交互系统、方法及装置

Publications (1)

Publication Number Publication Date
WO2017118075A1 true WO2017118075A1 (zh) 2017-07-13

Family

ID=55674697

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/098568 WO2017118075A1 (zh) 2016-01-04 2016-09-09 人机交互系统、方法及装置

Country Status (3)

Country Link
US (1) US10585488B2 (zh)
CN (1) CN105487673B (zh)
WO (1) WO2017118075A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112825013A (zh) * 2019-11-20 2021-05-21 百度在线网络技术（北京）有限公司 终端设备的控制方法和装置
CN113764093A (zh) * 2021-08-18 2021-12-07 上海电气集团股份有限公司 混合现实显示设备及其手术信息处理方法、存储介质

Families Citing this family (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105487673B (zh) * 2016-01-04 2018-01-09 京东方科技集团股份有限公司 一种人机交互系统、方法及装置
CN105975058A (zh) * 2016-04-26 2016-09-28 上海与德通讯技术有限公司 虚拟现实一体机的控制方法及装置
CN105975158A (zh) * 2016-05-11 2016-09-28 乐视控股（北京）有限公司 虚拟现实交互的方法及装置
CN107450715A (zh) * 2016-05-31 2017-12-08 大唐电信科技股份有限公司 一种基于手势识别的人机交互多功能腕带终端
CN106094989A (zh) * 2016-06-10 2016-11-09 北京行云时空科技有限公司 一种增强现实图像定位显示方法、装置和系统
CN106125918A (zh) * 2016-06-18 2016-11-16 深圳晨芯时代科技有限公司 一种虚拟现实装置和虚拟现实分享系统
CN106155311A (zh) * 2016-06-28 2016-11-23 努比亚技术有限公司 Ar头戴设备、ar交互系统及ar场景的交互方法
CN106200964B (zh) * 2016-07-06 2018-10-26 浙江大学 一种虚拟现实中基于移动轨迹识别进行人机交互的方法
CN106200956A (zh) * 2016-07-07 2016-12-07 北京时代拓灵科技有限公司 一种虚拟现实领域多媒体呈现和交互的方法
CN106155324B (zh) * 2016-07-13 2019-05-31 中国人民解放军海军医学研究所 一种操作水下武器装备的人机交互辅助方法及系统
CN106484102A (zh) * 2016-09-19 2017-03-08 惠州Tcl移动通信有限公司 一种根据手势识别用户操作动作的方法及系统
CN106896914A (zh) * 2017-01-17 2017-06-27 珠海格力电器股份有限公司 信息的转换方法和装置
US10261595B1 (en) * 2017-05-19 2019-04-16 Facebook Technologies, Llc High resolution tracking and response to hand gestures through three dimensions
CN107272890A (zh) * 2017-05-26 2017-10-20 歌尔科技有限公司 一种基于手势识别的人机交互方法和装置
CN107340965A (zh) * 2017-06-28 2017-11-10 丝路视觉科技股份有限公司 桌面显示设备及其控制方法、待识别物体及其识别方法
WO2019100247A1 (zh) * 2017-11-22 2019-05-31 华为技术有限公司 应用于虚拟现实的图像显示方法、装置、设备及系统
CN108460354B (zh) * 2018-03-09 2020-12-29 深圳臻迪信息技术有限公司 无人机控制方法、装置、无人机及系统
US10635895B2 (en) 2018-06-27 2020-04-28 Facebook Technologies, Llc Gesture-based casting and manipulation of virtual content in artificial-reality environments
CN109254650B (zh) * 2018-08-02 2021-02-09 创新先进技术有限公司 一种人机交互方法和装置
CN109710066B (zh) * 2018-12-19 2022-03-25 平安普惠企业管理有限公司 基于手势识别的交互方法、装置、存储介质和电子设备
CN109672866A (zh) * 2018-12-27 2019-04-23 高腾飞 一种监控设备远程安全控制方法及系统
CN111913639B (zh) * 2019-05-07 2022-01-28 广东虚拟现实科技有限公司 虚拟内容的交互方法、装置、系统、终端设备及存储介质
US11334212B2 (en) * 2019-06-07 2022-05-17 Facebook Technologies, Llc Detecting input in artificial reality systems based on a pinch and pull gesture
CN110442238A (zh) * 2019-07-31 2019-11-12 腾讯科技（深圳）有限公司 一种确定动态效果的方法及装置
CN110780734B (zh) * 2019-09-25 2023-11-03 深圳清元文化科技有限公司 一种基于雷达的手势交互ar投影方法及装置
CN111377169A (zh) * 2020-01-23 2020-07-07 杭州睿杨环境科技有限公司 一种基于人工智能技术的多功能垃圾箱
CN111627039A (zh) * 2020-05-09 2020-09-04 北京小狗智能机器人技术有限公司 一种基于图像识别的交互系统及交互方法
CN111470216A (zh) * 2020-05-25 2020-07-31 孙科航 基于手势识别控制的分类垃圾箱开盖装置
CN111627097B (zh) * 2020-06-01 2023-12-01 上海商汤智能科技有限公司 一种虚拟景物的展示方法及装置
CN112578987A (zh) * 2020-12-25 2021-03-30 广州壹创电子科技有限公司 屏外交互式触摸一体机及其交互方法
CN113419636B (zh) * 2021-08-23 2021-11-30 北京航空航天大学 虚拟维修中手势识别方法及工具自动匹配方法
CN114706486A (zh) * 2022-04-26 2022-07-05 四川大学 基于手势识别的混合现实工业控制方法及装置

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103295029A (zh) * 2013-05-21 2013-09-11 深圳Tcl新技术有限公司 手势控制终端的交互方法及装置
CN103576840A (zh) * 2012-07-24 2014-02-12 上海辰戌信息科技有限公司 基于立体视觉的手势体感控制系统
US8928590B1 (en) * 2012-04-03 2015-01-06 Edge 3 Technologies, Inc. Gesture keyboard method and apparatus
CN105045398A (zh) * 2015-09-07 2015-11-11 哈尔滨市一舍科技有限公司 一种基于手势识别的虚拟现实交互设备
CN105487673A (zh) * 2016-01-04 2016-04-13 京东方科技集团股份有限公司 一种人机交互系统、方法及装置
CN205485916U (zh) * 2016-01-04 2016-08-17 京东方科技集团股份有限公司 一种人机交互系统

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103064514A (zh) * 2012-12-13 2013-04-24 航天科工仿真技术有限责任公司 沉浸式虚拟现实系统中的空间菜单的实现方法
US9459697B2 (en) * 2013-01-15 2016-10-04 Leap Motion, Inc. Dynamic, free-space user interactions for machine control
WO2015123771A1 (en) * 2014-02-18 2015-08-27 Sulon Technologies Inc. Gesture tracking and control in augmented and virtual reality
CN204463032U (zh) * 2014-12-30 2015-07-08 青岛歌尔声学科技有限公司 一种3d场景中输入手势的系统和虚拟现实头戴设备
CN105068649A (zh) * 2015-08-12 2015-11-18 深圳市埃微信息技术有限公司 基于虚拟现实头盔的双目手势识别装置及方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8928590B1 (en) * 2012-04-03 2015-01-06 Edge 3 Technologies, Inc. Gesture keyboard method and apparatus
CN103576840A (zh) * 2012-07-24 2014-02-12 上海辰戌信息科技有限公司 基于立体视觉的手势体感控制系统
CN103295029A (zh) * 2013-05-21 2013-09-11 深圳Tcl新技术有限公司 手势控制终端的交互方法及装置
CN105045398A (zh) * 2015-09-07 2015-11-11 哈尔滨市一舍科技有限公司 一种基于手势识别的虚拟现实交互设备
CN105487673A (zh) * 2016-01-04 2016-04-13 京东方科技集团股份有限公司 一种人机交互系统、方法及装置
CN205485916U (zh) * 2016-01-04 2016-08-17 京东方科技集团股份有限公司 一种人机交互系统

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112825013A (zh) * 2019-11-20 2021-05-21 百度在线网络技术（北京）有限公司 终端设备的控制方法和装置
CN113764093A (zh) * 2021-08-18 2021-12-07 上海电气集团股份有限公司 混合现实显示设备及其手术信息处理方法、存储介质

Also Published As

Publication number Publication date
CN105487673A (zh) 2016-04-13
CN105487673B (zh) 2018-01-09
US20180101237A1 (en) 2018-04-12
US10585488B2 (en) 2020-03-10

Similar Documents

Publication Publication Date Title
WO2017118075A1 (zh) 人机交互系统、方法及装置
US20220326781A1 (en) Bimanual interactions between mapped hand regions for controlling virtual and graphical elements
US9651782B2 (en) Wearable tracking device
KR101844390B1 (ko) 사용자 인터페이스 제어를 위한 시스템 및 기법
US11170580B2 (en) Information processing device, information processing method, and recording medium
CN107390863B (zh) 设备的控制方法及装置、电子设备、存储介质
JP6259545B2 (ja) ３ｄシーンでジェスチャーを入力するシステム及び方法
US9442571B2 (en) Control method for generating control instruction based on motion parameter of hand and electronic device using the control method
WO2021227628A1 (zh) 一种电子设备及其交互方法
US10564712B2 (en) Information processing device, information processing method, and program
CN108027654B (zh) 输入设备、输入方法和程序
US20120229509A1 (en) System and method for user interaction
WO2013139181A1 (zh) 一种用户交互系统和方法
WO2012119371A1 (zh) 一种用户交互系统和方法
CN105068646B (zh) 终端的控制方法和系统
WO2017057106A1 (ja) 入力装置、入力方法、及びプログラム
WO2021097600A1 (zh) 一种隔空交互方法、装置和设备
CN109839827B (zh) 一种基于全空间位置信息的手势识别智能家居控制系统
JP6516464B2 (ja) ウェアラブル検索システム
CN104077784B (zh) 提取目标对象的方法和电子设备
Chen et al. Lisee: A headphone that provides all-day assistance for blind and low-vision users to reach surrounding objects
CN103713387A (zh) 电子设备和采集方法
CN104156138B (zh) 拍摄控制方法及拍摄控制装置
CN113096193A (zh) 三维体感操作的识别方法、装置和电子设备
CN111766942A (zh) 基于智能戒指的输入方法、系统、智能戒指及移动设备

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 15519689

Country of ref document: US

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16883205

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16883205

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 21.06.2019)

122 Ep: pct application non-entry in european phase

Ref document number: 16883205

Country of ref document: EP

Kind code of ref document: A1