CN109963034A - The capture of intelligent electronic device gesture and identification technology based on grip hand detection - Google Patents
The capture of intelligent electronic device gesture and identification technology based on grip hand detection Download PDFInfo
- Publication number
- CN109963034A CN109963034A CN201910154756.4A CN201910154756A CN109963034A CN 109963034 A CN109963034 A CN 109963034A CN 201910154756 A CN201910154756 A CN 201910154756A CN 109963034 A CN109963034 A CN 109963034A
- Authority
- CN
- China
- Prior art keywords
- electronic device
- hand
- camera
- intelligent electronic
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M1/00—Substation equipment, e.g. for use by subscribers
- H04M1/02—Constructional features of telephone sets
- H04M1/0202—Portable telephone sets, e.g. cordless phones, mobile phones or bar type handsets
- H04M1/026—Details of the structure or mounting of specific components
- H04M1/0264—Details of the structure or mounting of specific components for a camera module assembly
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M1/00—Substation equipment, e.g. for use by subscribers
- H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
- H04M1/724—User interfaces specially adapted for cordless or mobile telephones
- H04M1/72448—User interfaces specially adapted for cordless or mobile telephones with means for adapting the functionality of the device according to specific conditions
- H04M1/72454—User interfaces specially adapted for cordless or mobile telephones with means for adapting the functionality of the device according to specific conditions according to context-related or environment-related conditions
Landscapes
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Human Computer Interaction (AREA)
- Computer Networks & Wireless Communication (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Provide a kind of intelligent electronic device and exchange method, intelligent electronic device is deployed with camera, the camera can capture the image of the grip hand of user's holding apparatus, the image of grip hand of the intelligent electronic device based on user obtained, the gesture of identification grip hand, the gesture of grip hand based on identification, intelligent electronic device execute corresponding control operation.The gesture of identification grip hand further includes the position of gripping finger around tracking electronic equipment, and using the movement for lifting, moving or tapping intelligent electronic device of different grip fingers as the information input of interaction.Compared with the information input mode based on touch screen of conventional mobile phone, the gesture interaction based on grip hand can enrich the input mode of mobile phone, and some interactions are become more convenient.
Description
Technical field
The present invention relates generally to the input technology and interaction technique of smart electronics portable equipment, more particularly to by setting
Mirror assembly is set to enable mobile phone to capture new gesture and construct the technology of binocular vision system.
Background technique
The field range of camera is limited and fixed on mobile phone at present, and such as front camera can only obtain right above mobile phone
Image information in 60*80 degree or so spatial dimension, the existing camera of (2) mobile phone is essentially all monocular cam, and monocular is taken the photograph
As head can only obtain the RGB information of its object within sweep of the eye, its three-dimensional information can not be obtained.
Due to the defect of the existing existing camera of mobile phone, when we are not available mobile phone and obtain user and use mobile phone naturally
Hand information, it is many to be ignored to the valuable hand information of mobile phone interaction.
The input channel on mobile phone is also limited at present, and most information is all the number read by capacitive screen of mobile phone
According to, i.e., directly contact and generating between the hand with touch screen of user, thus will appear it is some cumbersome or operate not oneself
Phenomena such as right.
Summary of the invention
In view of the foregoing, the present invention is proposed.
According to an aspect of the invention, there is provided a kind of intelligent electronic device, is deployed with camera, the camera energy
The image intelligent electronic equipment of the grip hand of user's holding apparatus is enough captured based on the image of the grip hand of user obtained, is known
Not Chi Wo hand gesture, the gesture of the grip hand based on identification, intelligent electronic device executes corresponding control operation.
Optionally, the gesture of the identification grip hand includes the position of gripping finger around tracking electronic equipment, and with
Information input of the movement for lifting, moving or tapping intelligent electronic device of difference grip finger as interaction.
Optionally, the grip gesture model includes the gesture that hand-held intelligent electronic equipment is taken pictures, and is referred to as gesture of taking pictures,
When intelligent electronic device, which recognizes, takes pictures gesture, intelligent electronic device starts the application taken pictures automatically, and automatically snaps one
Photo.
Optionally, intelligent electronic device also can also operate to: identification user is held with which hand；Based on knowledge
Not as a result, being adjusted to graphic user interface layout, enable the finger of the grip hand of user is preceding compared to adjustment more to hold
It changes places and clicks target component.
Optionally, camera is at the edge of intelligent electronic device.
Optionally, camera is the fish-eye camera below screen.
Optionally, intelligent electronic device further includes the infrared illumination being used in combination with camera and infrared fileter, to increase
Big signal-to-noise ratio.
Optionally, camera is depth camera.
Optionally, intelligent electronic device further includes the optical reflection unit for favouring the screen arrangement of intelligent electronic device,
The light reflection unit can light of the reflected parallel in electronic equipment screen surface capture it by camera, to be held in user
When holding electronic equipment, camera can capture the image of hand of above the screen, user when holding electronic equipment；Intelligence
Hand images of the electronic equipment based on user obtained identify the hand motion and/or posture of user at this time, as user's
Information is inputted, and is interacted with user.
Optionally, the light reflection unit is one of eyeglass, prism, convex mirror and more eyeglasses or a combination thereof.
Optionally, the camera is wide-angle camera.
Optionally, the wide-angle camera is placed in center Screen, and optical axis direction is perpendicular to electronic device touchscreen, tool
There are 170 degree to 190 degree visual angles.
Optionally, the camera is liftable camera and/or adjustable angle camera.
Optionally, the field range of the liftable camera and/or adjustable angle camera be its field range with
Be parallel to electronic equipment plane longitudinal axis sight be zero degree line, in the normal plane of the lateral shaft of electronic equipment plane at least-
40 degree to 40 degree of range.
Optionally, the intelligent electronic device is smart phone, intelligent vehicle-carried electronic equipment, appointing in Intelligent flat computer
One.
According to another aspect of the present invention, it provides a kind of intelligent electronic device and is deployed with camera, the camera shooting
Head can capture the image of the grip hand of user's holding apparatus, and the man-machine interaction method includes: based on user obtained
Hold the image of hand, the gesture of identification grip hand, the gesture of the grip hand based on identification, the corresponding control behaviour of electronic equipment execution
Make.
Optionally, intelligent electronic device has camera and favours the light reflection dress of electronic equipment screen surface layout
It sets, the man-machine interaction method, which includes: camera system, which captures above screen, user when user holds electronic equipment, makes
The image of hand when with electronic equipment；Hand images of the intelligent electronic device based on user obtained identify user at this time
Hand motion and/or posture, as the input information of user, and based on this input information interacted with user.
According to another aspect of the present invention, a kind of intelligent electronic device is provided, there is sensor, the sensor can
The information of hand of above screen, user when using electronic equipment is sensed, wherein intelligent electronic device processing passes
The data of sensor sensing identify that the movement between both hands finger is gone forward side by side as user for the gesture input of intelligent electronic device
The corresponding control operation of row.
Optionally, the sensor is the camera system being mounted on intelligent electronic device, can be captured above screen
, the image of the hand that user is when using electronic equipment.
Optionally, the image of hand when further including based on user obtained grip electronic equipment, identifies user at this time
Grip state under gesture, as the input information of user, and interacted with user, wherein intelligent electronic device carry out it is following
At least one of in gesture identification: intelligent electronic device identifies single hand grip posture, and identification another hand to grip hand
Touch action, as the gesture input with intelligent electronic device, to carry out the interaction with user；Intelligent electronic device identifies both hands
The movement between posture, and identification both hands thumb is held, as the gesture input for being directed to intelligent electronic device, to carry out and use
The interaction at family.
Optionally, the camera system include front camera and favour electronic equipment screen surface layout eyeglass or
Prism, the eyeglass or prism reflected parallel capture it by front camera in the light on electronic equipment screen surface, preposition
Camera captures the light through eyeglass or prism reflection, thus when capturing user's grip electronic equipment, above screen, use
The image of hand of family when using electronic equipment；Intelligent electronic device holds hand when electronic equipment based on user obtained
The image in portion identifies the gesture under the grip state of user at this time, as the input information of user, and interacts with user, wherein
Intelligent electronic device carries out at least one intelligent electronic device identification single hand grip posture in following gesture identifications, and identification
Another hand is to the touch action for holding hand, as the gesture input with intelligent electronic device, to carry out the interaction intelligence with user
Energy electronic equipment identifies the movement between both hands grip posture, and identification both hands thumb, as the hand with intelligent electronic device
Gesture input, to carry out the interaction with user.
Optionally, intelligent electronic device refers to by identification side outstanding four and appears in above electronic equipment or side
A thumb, to identify single hand grip posture.
Optionally, intelligent electronic device is referred to above root and screen by the thumb that identification appears in intelligent electronic device two sides
Two thumb fingers, to identify both hands grip posture.
Optionally, in the case where single hand grip, another hand includes: finger button and hand to the touch action of grip hand
Refer to slider bar gesture, wherein in finger button gesture, to hold the finger of hand as button, another hand touches it,
In finger slider bar gesture, to hold the finger of hand as slider bar, another hand slides on or clicks.
Optionally, the movement between the both hands thumb includes one or more of following: thumb touches；Thumb wheel
Turn；Thumb presses certain path movement respectively；Thumb is acted after touching by certain path.
According to another aspect of the present invention, a kind of man-machine interaction method of intelligent electronic device is provided, smart electronics are set
Standby to have sensor, the sensor can capture the image of hand of above screen, user when using electronic equipment,
Wherein the man-machine interaction method includes: the movement between the intelligent electronic device identification both hands finger, is directed to as user
The gesture input of intelligent electronic device, and carry out corresponding control operation.
Optionally, the sensor is the camera system being mounted on intelligent electronic device, can be captured above screen
, the hand images that user is when using electronic equipment.
Optionally, man-machine interaction method further includes hand images when holding electronic equipment based on user obtained, is known
The not gesture under the grip state of user at this time interacts wherein as the input information of user, and with user, identification user at this time
Grip state under gesture include carry out in following gesture identifications at least one of: identification single hand grip posture, and identification
Another hand is to the touch action for holding hand, as the gesture input with intelligent electronic device, to carry out the interaction with user；Know
Other both hands grip posture, and identification both hands thumb between movement, as be directed to intelligent electronic device gesture input, into
The capable interaction with user.
Optionally, the camera system includes front camera and the mirror for favouring intelligent electronic device screen surface arrangement
Piece or prism, the eyeglass or prism reflected parallel capture it by front camera in the light on electronic equipment screen surface,
Front camera captures the light through eyeglass or prism reflection, thus when capturing user's grip electronic equipment, above screen
, the image of the hand that user is when using electronic equipment.
Optionally, intelligent electronic device referred to by identification side outstanding four and appeared in above intelligent electronic device or
One thumb of side, to identify single hand grip posture.
Optionally, intelligent electronic device is referred to above root and screen by the thumb that identification appears in intelligent electronic device two sides
Two thumb fingers, to identify both hands grip posture.
Optionally, in the case where single hand grip, another hand includes: finger button and hand to the touch action of grip hand
Refer to slider bar gesture, wherein in finger button gesture, to hold the finger of hand as button, another hand touches it,
In finger slider bar gesture, to hold the finger of hand as slider bar, another hand slides on or clicks.
Optionally, the movement between the both hands thumb includes one or more of following: thumb touches thumb rotation；
Thumb presses certain path movement respectively；Thumb is acted after touching by certain path.
According to another aspect of the present invention, a kind of intelligent electronic device is provided, there is front camera and favours electricity
The light reflection unit of sub- device screen surface layout, the light reflecting device make the light for being parallel to electronic equipment screen direction
It can be reflected into front camera through it, reflection of the light a little issued from object via prism, the screen of portable equipment, meeting
Enter camera by two optical paths, thus generates two virtual cameras, build virtual binocular camera, and then obtain
Obtain the space three-dimensional information of object.
Optionally, intelligent electronic device is further equipped with infrared light-emitting device, and the camera is infrared camera.
Optionally, the light reflecting device is prism.
Optionally, the light reflecting device is plane mirror or convex mirror.
Optionally, the binocular camera can capture the left and right lateral at least each 5 using electronic equipment bottom as starting point
Centimetre, the image of the object at least 10 cm ranges longitudinally upward.
Optionally, one of two optical paths are that the light that object a little issues directly is mapped to light reflecting device, are then reflected through light
Enter front camera after device reflection；Another light issued for the object point in another optical path first passes around electronic equipment
After screen reflection, it is mapped to light reflecting device, front camera is entered after then being reflected by light reflecting device.
Optionally, there is relatively darker region band, referred to as blanking bar, the rib in the image that front camera captures
Mirror can be adjusted relative to the position of front camera, to reduce the range of blanking bar.
Optionally, there is the opposite darker region in other regions, referred to as blanking bar in the image that front camera captures,
Intelligent electronic device removes blanking bar using luminance compensation method before calculating Stereo Vision.
Optionally, image of the intelligent electronic device based on acquisition identifies the object in image, and the knot based on identification
Fruit, to interact.
Optionally, in the case where recognizing object is one-pen, the angle between pen body and electronic curtain surface is estimated,
And based on the angle estimated, to control the interactive operation of intelligent electronic device.
Optionally, for the binocular RGB image of capture, following processing are executed to carry out gesture identification:
(1) it is corrected processing, obtains standardized monocular RGB image；
(2) hand skin mask image is obtained using skin color segmentation method；
(3) two standardized monocular RGB images are based on, individual element calculates depth, obtains depth map；
(4) skin mask image and depth map are combined, the segmented image of hand region is obtained；
(5) segmented image based on obtained hand region carries out gesture identification.
Optionally, the correction process includes carrying out color school using pixel color equation Output=Input*R+L
Just, wherein Output be output image in each pixel color, Input be input image in each pixel face
Color, R are reflection factor, are only dependent upon the physical property of screen surface, and L is the self-luminous of electronic equipment.
Optionally, it is zero that the L, which is arranged, to acquire the image of white metope, fitting by using the intelligent electronic device
Obtain parameter R.
Optionally, the skin color segmentation algorithm includes two modules, and a module utilizes the tone and saturation degree in image
Threshold value to divide skin area, another module dynamically calibrates these threshold values every predetermined number of frames.
According to another aspect of the present invention, a kind of man-machine interaction method of smart electronics portable equipment, intelligence electricity are provided
Sub- portable equipment has front camera and favours the light reflecting device of electronic equipment screen surface layout, and light reflecting device is
Planar optics or prism, the man-machine interaction method include that reflection unit makes the luminous energy for being parallel to electronic equipment screen direction
Enough to be reflected into front camera through it, the light a little issued from object passes through via the reflection of prism, the screen of portable equipment
Two optical paths enter camera, obtain this two flat images of two flat image portable devices handles, obtain depth information knot
It closes depth information and carries out object identification, the object based on identification carries out human-computer interaction.
Optionally, smart electronics portable equipment is further equipped with infrared light-emitting device, and the camera is infrared camera.
Optionally, the light reflecting device is prism.
Optionally, the light reflecting device is plane mirror or convex lens.
Optionally, the binocular camera can capture the left and right lateral at least each 5 using electronic equipment bottom as starting point
Centimetre, the image of the object at least 10 cm ranges longitudinally upward.
Optionally, one of two optical paths are that the light that object a little issues directly is mapped to light reflecting device, are then reflected through light
Enter front camera after device reflection；Another light issued for the object point in another optical path first passes around electronic equipment
After screen reflection, it is mapped to light reflecting device, front camera is entered after then being reflected by light reflecting device.
Optionally, there is relatively darker region band, referred to as blanking bar, the rib in the image that front camera captures
Mirror can be adjusted relative to the position of front camera, to reduce the range of blanking bar.
Optionally, there is the opposite darker region in other regions, referred to as blanking bar in the image that front camera captures,
Smart electronics portable equipment removes blanking bar using luminance compensation method before calculating Stereo Vision.
Optionally, image of the intelligent electronic device based on acquisition identifies the object in image, and the knot based on identification
Fruit, to interact.
Optionally, in the case where recognizing object is one-pen, the angle between pen body and electronic curtain surface is estimated,
And based on the angle estimated, to control the interactive operation of intelligent electronic device.
Optionally, man-machine interaction method further includes the hand motion and/or posture in conjunction with depth information identification user, as
The input information of user, and interacted based on this input information with user.
It optionally, further include specific two monoculars for capture in the case where captured image is monocular RGB image
RGB image executes following processing to carry out gesture identification: (1) being corrected processing, obtain standardized monocular RGB image；
(2) using skin color segmentation method obtain hand skin mask image (3) be based on two standardized monocular RGB images, one by one as
Element calculates depth, obtains depth map；(4) skin mask image and depth map are combined, the segmented image of hand region is obtained；(5)
Based on the segmented image of obtained hand region, gesture identification is carried out.
Optionally, the correction process includes carrying out color school using pixel color equation Output=Input*R+L
Just wherein Output is the color of each pixel in the image of output, and Input is the face of each pixel in the image of input
Color, R are reflection factor, are only dependent upon the physical property of screen surface, and L is the self-luminous of electronic equipment.
Optionally, it is zero that the L, which is arranged, to acquire the image of white metope, fitting by using the intelligent electronic device
Obtain parameter R.
Optionally, the skin color segmentation algorithm includes two modules, and a module utilizes the tone and saturation degree in image
Threshold value to divide skin area, another module dynamically calibrates these threshold values every predetermined number of frames.
According to another aspect of the present invention, a kind of intelligent electronic device includes a tangible surface, has sensor,
The sensor can capture hand images when contacting with tangible surface, wherein intelligent electronic device identification is in hand
The state of hand when touching tangible surface touches the concrete position on tangible surface including identification hand, identification is touched and can be touched
Touch surface is angle one or more of of which root finger, finger relative to tangible surface.
Optionally, wherein the concrete position of identification hand includes identifying finger tip, finger pulp, great thenar, one in articulations digitorum manus
Or it is multiple, different finger positions indicates to carry out different objects corresponding to the same movement of same position on tangible surface
Operation.
Optionally, the sensor includes camera, and the concrete position of hand is detected, camera captured image is utilized
Signal, the image procossing based on Morphology observation are completed.
Optionally, wherein identifying that related finger be which root finger includes: identification thumb, index finger, middle finger, the third finger and small
The one or more of finger, different fingers indicate to carry out different operation to this interface target to the same movement of same interface target.
Optionally, it for using the identification of finger, is completed using deep neural network.
Optionally, wherein identification finger relative to angle include: identification finger relative to tangible surface institute in 0
To the various angles in 90 degree of ranges, and identify click of these fingers to tangible surface different angle, or after click
Finger angular adjustment is used as information input.
Optionally, detect finger touch loudness/brightness regulation button in the case where, based on detect finger relative to
The variation of the angle of screen adjusts loudness/brightness size.
Optionally, the identification finger includes: to pass through seat according to the position of screen capacitance signal relative to the angle of screen
Mark, which converts, determines the position of hand clicking point in the picture, and then determines click on area in the picture, uses linear regression prediction
Method, the point cloud of fingertip area in depth map is fitted, so determine click finger angle, the click on area packet
Include the specific region above click location and click location.
Optionally, the concrete position of hand is detected, in conjunction with screen capacitance signal and camera captured image signal, base
It is completed in the image procossing of Morphology observation.
Optionally, electronic equipment is one of smart phone, touch tablet, mobile unit.
The scheme of one embodiment of the invention can utilize one piece of mirror (including plane mirror, prism or convex lens etc.), change
Become the field range of the existing camera of mobile phone.The field range of camera is limited and fixed on conventional mobile phone, such as front camera,
The image information right above mobile phone in 60*80 degree or so spatial dimension can only be obtained, be not available mobile phone acquisition user makes naturally
Hand information when with mobile phone, it is many to be ignored to the valuable hand information of mobile phone interaction.This programme is by increasing by one
The mirror fitting and corresponding algorithm of low cost, change the field range of front camera, so that by the reflection of mirror surface, it is existing
Camera can capture user (naturally use mobile phone when) hand images information, widened the view of cell-phone camera system significantly
Wild range.
The scheme of another embodiment of the present invention provides inexpensive (only needing an one piece of mirror) binocular vision for mobile phone
System, two optical paths generated by this block mirror and the existing camera cooperation of mobile phone, constructs a virtual binocular camera shooting
Head restores the three-dimensional information of object in space.
Another embodiment of the present invention additionally provides based on the interaction gesture for holding hand, with conventional mobile phone based on touch screen
Information input mode is compared, and the gesture interaction based on grip hand can enrich the input mode of mobile phone, and some interactions are become more
It is convenient to add.Further, since self perception of human body, user can easily position the physical feeling of oneself, therefore user can be
The privileged site that grip hand is shone in the case where without visual attention location interacts.This interactive mode of the embodiment of the present invention is more
Add the use habit met when user uses mobile phone naturally, do not need purposely lift using hand, makes in space not certainly
Right gesture.Actual user experience experiment shows that these interaction design modes can be received by user well, together
When have interest, learnability and convenience.
Detailed description of the invention
From the detailed description with reference to the accompanying drawing to the embodiment of the present invention, these and/or other aspects of the invention and
Advantage will become clearer and be easier to understand, in which:
Fig. 1, catching for the mirror camera that mobile phone screen is placed is favoured Fig. 2 shows utilization according to an embodiment of the present invention
Range is obtained to be changed to cover the schematic diagram in the space close to mobile phone screen parallel with mobile phone screen.
Fig. 3 show the man-machine interaction method of the mobile phone according to an embodiment of the present invention for being placed with tilting mirror one shows
Example property operating process.
Fig. 4 shows the schematic diagram that wide-angle camera in mobile phone is placed in center Screen, and Fig. 5 shows view in this case
The schematic diagram of wild range.
Fig. 6 shows the schematic diagram of the case where installation liftable camera at the top of mobile phone.
Fig. 7 shows the schematic diagram for the case where adjustable camera is installed at mobile phone edge
Fig. 8 shows the field range pair of traditional front camera of camera installation way and mobile phone in Fig. 6 and Fig. 7
The schematic diagram of ratio.
Fig. 9 shows user's index finger according to an embodiment of the invention and clicks mobile phone side, occur in screen cursor with
The schematic diagram of a scenario for facilitating user to click distant objects.
Figure 10 is shown when user picks up mobile phone with the gesture taken pictures, camera app should automatic identification open, and it is automatic
Shoot the schematic diagram of a scenario of a photo.
Figure 11 is shown after identifying user and being held with which hand, so that it may is adjusted, be made to UI layout
Obtain the schematic diagram that user preferably clicks target component.
Figure 12 shows the application schematic diagram of finger slider bar and finger button.
Figure 13 shows both hands thumb gesture schematic diagram when hands grasping mobile phone.
Figure 14 shows mirror and mobile phone has the optical principle schematic diagram that virtual binocular camera is built in camera cooperation.
Figure 15 is shown will appear relatively in the image that mobile phone front camera captures using prism
The schematic diagram of darker region band.
Specific embodiment
In order to make those skilled in the art more fully understand the present invention, with reference to the accompanying drawings and detailed description to this hair
It is bright to be described in further detail.
Before introduction, the meaning herein in relation to term is explained.
Mirror, mirror herein is broader sense, for the device with light reflection function, such as planar optics, trigone
Mirror, convex lens etc..
Hand skin mask image, only hand region exists in image, the image that other backgrounds are removed.With other figures
When piece combines, it can distinguish and cover in undesired part.
" hand of above screen, user when using electronic equipment " herein, refers to above screen in 5 centimetres
Hand of user when using electronic equipment, the front camera of such hand conventional mobile phone be basic shooting less than
's.
One, the human-computer interaction based on grip hand
Gesture interaction on existing mobile phone does not consider user's posture feature when using mobile phone, they need purposely
It will be lifted using hand, and make unnatural gesture in space.
Inventor is the body part nearest apart from mobile phone in view of grip hand, and the interaction based on grip hand is more in line with use
Family uses behavioural habits when mobile phone naturally, can enrich the information input mode of mobile phone, and there have to be more new convenient and natural
Interaction may.
Inventor is further contemplated that mobile phone holds hand to capture, and traditional sensor uses not convenient enough.Want
Hand information of user when using mobile phone is perceived, current techniques are mostly using mobile phone built-in sensors, such as acceleration sensing
Device, gravity sensor etc., some research associations external ultrasonic sensor on mobile phone carry out acquisition of information.These methods can only
Obtain specific, simple hand information, such as hand whether close to mobile phone, hand moving direction, whether rock mobile phone, can not obtain
The information of hand must be held.
According to one embodiment of present invention, the interaction technique based on grip hand is proposed, installing on mobile phone can catch
The camera for obtaining grip hand image of user when using mobile phone, captures the image of grip hand, with computer vision
Technology obtains the gesture information of grip hand, can not only identify grip, additionally it is possible to track the position that finger is held around mobile phone
It sets.By obtaining these information of grip hand, mobile phone executes corresponding control operation.This interactive mode based on grip hand,
The information input mode of mobile phone can be enriched, many operations can also be allowed to become more intelligent and quick.
About the form of camera, it can be existing camera and be furnished with the mirror additionally configured, or be fish-eye camera.
According to an embodiment of the present invention, using one piece of mirror being tiltedly placed on above the existing camera of mobile phone, change existing
The capture space of camera.Therefore, which can obtain user (when naturally using mobile phone) hand by the reflection of mirror surface
Portion's image information, the capture range of camera is changed to cover the sky close to mobile phone screen parallel with mobile phone screen at this time
Between, as shown in Figure 1 and Figure 2.Front camera in existing mobile phone is mainly used for capturing the forward or backward subject image of mobile phone,
And operation of the human hand to mobile phone cannot be captured, such as the hand for holding mobile phone that cannot be captured, opponent in one's hands can not be captured
The operation of machine, field range is about using vertical mobile phone line as 80 degree of ranges of axis, as shown in the upper figure in Fig. 5；It compares, this
Inventive embodiments by mobile phone side and mobile phone screen it is inclined arrangement one can reflected parallel in mobile phone screen surface
Light captures it by camera, so that camera, which can capture, is related to side parallel with mobile phone screen when user holds mobile phone
The image of hand of upward light, above screen, user when using mobile phone, and the gesture based on this identification user,
It is fed back to be based on coming gesture.
One exemplary operation of the man-machine interaction method of the mobile phone according to an embodiment of the present invention for being placed with tilting mirror
Process is as shown in Figure 3: in step 110, capture user holds hand images when mobile phone；In the step 120, using computer
The algorithm of vision identifies gesture；In step 130, mobile phone executes corresponding operation.
Preferably, the position of camera is in mobile phone edge (including at the top of mobile phone and side).Alternatively, camera may be used also
With the various positions such as in mobile phone surface, the purpose finally realized is can to capture user's image of both hands when using mobile phone to believe
Breath.The visual field of camera may be it is fixed, be specifically used to shoot user's both hands；It may also be able to carry out adjustment, in needs
When the visual field is adjusted on user's both hands.Several possible situations are listed below:
(1) camera is mounted in mobile phone screen, and the camera of wide-angle can capture big portion of user when using mobile phone
Divide hand motion:
The wide-angle camera is placed in center Screen, as shown in figure 4, its field range is longitudinal to be parallel to mobile phone plane
The sight of axis is the range of zero degree line, 0 degree in the normal plane of the lateral shaft of mobile phone plane to 180 degree, below in Fig. 5
Figure shown in.
(2) liftable camera being mounted on mobile phone, Fig. 6 show the scalable camera being mounted at the top of mobile phone and show
It is intended to.
(3) it is mounted on the adjustable camera at mobile phone edge, Fig. 7 shows the adjustable camera being mounted at the top of mobile phone
Schematic diagram.
The field range that Fig. 8 shows traditional front camera of above-mentioned (2), (3) kind mounting means and mobile phone compares
Schematic diagram, wherein the upper figure in Fig. 8 is the field range schematic diagram of mobile phone tradition front camera, the following figure in Fig. 8 be this
The field range schematic diagram of mounting means is planted in (2), (3) of inventive embodiments.
In one example, the infrared illumination being used in combination with camera and infrared is also configured on smart electronics portable equipment
Optical filter, to increase signal-to-noise ratio.
In one example, it there also is provided depth camera on smart electronics portable equipment, so as to obtain depth letter
Breath, it is subsequent that the method for obtaining depth information to configurable deep camera is described in detail.
In one example, smart electronics portable equipment is configured with the screen arrangement for favouring smart electronics portable equipment
Optical reflection unit, the light reflection unit can light of the reflected parallel in electronic equipment screen surface catch it by camera
It catches, thus when user holds electronic equipment, camera can capture above screen, user when using electronic equipment
Hand images；Hand images of the smart electronics portable equipment based on user obtained, identification at this time the hand motion of user and/
Or posture, as the input information of user, and interacted with user.
Optionally, the light reflection unit is one of eyeglass, prism, convex mirror and more eyeglasses or a combination thereof.
Optionally, the camera is wide-angle camera.
Optionally, the wide-angle camera is placed in center Screen, and optical axis direction is perpendicular to electronic device touchscreen, tool
There are 170 degree to 190 degree visual angles.
Optionally, the camera is liftable camera and/or adjustable angle camera.
Optionally, the field range of the liftable camera and/or adjustable angle camera be its field range with
Be parallel to electronic equipment plane longitudinal axis sight be zero degree line, in the normal plane of the lateral shaft of electronic equipment plane at least-
40 degree to 40 degree of range.
Optionally, the smart electronics portable equipment is smart phone, in intelligent vehicle-carried electronic equipment, Intelligent flat computer
Either one or two of.
The hand information captured by camera, can not only identify grip, additionally it is possible to which identification is instantaneous or holds
Continuous grip, the position including holding finger around tracking mobile phone, so as to there is following application:
(1) finger position and the movement for obtaining grip hand, when user's single hand grip mobile phone, his hand to hold mobile phone
Referring to still has mobile space and flexibility.The movement for lifting, moving or tapping mobile phone of different fingers can be used as friendship
Mutual information input.Example is as follows: user's index finger clicks mobile phone side, cursor occurs in screen to facilitate user to distant objects
It is clicked, as shown in Figure 9.
(2) identification grip gesture.Holding gesture can be many valuable inputs of mobile phone belt, and such as various app's is quick
Unfolding mode: when user picks up mobile phone with the gesture taken pictures, camera app should automatic identification open, and automatically snap one
Photo, as shown in Figure 10.
(3) identification user holds hand.The screen size of current smart phone is often unsuitable for singlehanded use, especially in point
When hitting the content of screen remotely.Such as: after identifying user and being held with which hand, so that it may be laid out and carry out to UI
Adjustment, so that user preferably clicks target component, such as Figure 11.
Two, based on the new gesture acted between both hands finger
According to an embodiment of the present invention, a kind of intelligent electronic device is provided, there is sensor, the sensor can
The information of hand of above screen, user when using electronic equipment is sensed, wherein intelligent electronic device processing passes
Movement between the data of sensor sensing, identification both hands finger, the gesture input of portable equipment is directed to as user, and carries out phase
The control operation answered.
Here sensor is able to detect finger position, for example, imaging sensor (camera) or capacitance sensor.
In one example, the sensor is the camera system being mounted on intelligent electronic device, can capture screen
The hand images of curtain top, user when using electronic equipment.
It should be noted that sensor is not limited to imaging sensor (camera), other types of sensing can also be
Device, such as touch the capacitance sensor on screen.
According to one embodiment of present invention, a kind of gesture is devised the characteristics of the grip on mobile phone acts based on hand
Interactive mode.
Grip movement is divided into single hand grip and both hands grip.
In single hand grip, grip hand is grasped in meeting Lock-in side outstanding four on mobile phone and refers to and appear in mobile phone
The thumb of top or side, using these grip hands outstanding as the touch component that can be incuded, by the touching to grip hand, with
Mobile phone interacts.Figure 12 shows the application schematic diagram of finger slider bar and finger button, a hand grip mobile phone, another
Hand is touched, this is the typical gestures with mobile phone interaction.In this case, it is believed that grip hand, which can be taken as, to be used for
The interface of touch.In Figure 12, left figure is the case where finger of grip hand is as button, and we term it finger buttons
(FingerButton), finger button allows user to click the finger of grip hand as the interactive mode of manipulation application using finger
(for example, switching brush color).Right figure in Figure 12 shows the case where finger of grip hand is as slider bar, we term it
FingerBar, finger slider bar allow user to be slided on the thumb of grip hand to provide input (example to one-dimensional control strip
Such as, volume is controlled).
The step of operation needed for both technologies of FingerButton and FingerBar all reduce the interactive mode of script
Suddenly, and the medium that can be used for mobile phone input is increased, to improve interactive efficiency.
In both hands grip, two hands are all grip hands, and both hands thumb can be appeared in simultaneously above mobile phone screen, we will
Touching (such as thumb touches) between both hands thumb and movement (such as thumb rotation) as the mode interacted with mobile phone, we
Referred to as Thumb-to-Thumb gesture, the easily and rapidly operation as the second view of pattern switching or triggering.Such as Figure 13
It is shown, illustrate the example usage of enhancing typewriting input interactive experience: when progress important information is filled in mail, Yong Huke
The another application page can be wished to consult to obtain telephone number or address.In current usage mode, user must be switched back into
A upper application program, effort write down these key message character strings and back to input pages, and process is relatively complicated.It utilizes
Thumb-to-Thumb gesture, once detecting two thumb contacts, the screen that upper one applies can be moved to and currently be answered by system
Upper one layer, so as to user can easily reference content, after attempting memory, he/her can unclamp two thumbs and return to
Current page continues text input.This provides a kind of highly effective and light weight that pattern switching is carried out on smart phone
Method.
Compared with touch screen, the gesture interaction based on grip hand can enrich the input mode of mobile phone, and some interactions are become
It is more convenient to obtain.Further, since self perception of human body, user can easily position the physical feeling of oneself, therefore user can
It is interacted with shining the privileged site of grip hand in the case where being not necessarily to visual attention location.
Compared with the gesture interaction on mobile phone, this interactive mode, which is more in line with user and uses naturally, uses habit when mobile phone
It is used, it does not need purposely lift using hand, makes unnatural gesture in space.
Further, it is also possible to the finger movement etc. when identifying grip position used in grip mobile phone, grip mobile phone, such as hand is in mobile phone
Extend the basic gestures such as click, sliding, scaling, the selection in plane；Finger of the hand above mobile phone screen touches, pinches fist, opens
Hand, finger move special actions such as (such as finger are drawn a circle) along particular path.
In one example, the camera system in intelligent and portable equipment includes front camera and favours electronic equipment screen
The eyeglass or prism of curtain surface layout, the eyeglass or prism reflected parallel make its quilt in the light on electronic equipment screen surface
Front camera captures, and front camera captures the light through eyeglass or prism reflection, so that capturing user holds electronics
When equipment, hand images of above screen, user when using electronic equipment；Intelligent electronic device is based on user obtained
Hand images when electronic equipment are held, identify the gesture under the grip state of user at this time, as the input information of user, and
It is interacted with user, wherein portable equipment carries out at least one in following gesture identifications: portable equipment identifies single-hand handling & grip position
Gesture, and another hand is identified to the touch action of grip hand, as the gesture input with portable equipment, to carry out with user's
Interaction；Portable equipment identifies the movement between both hands grip posture, and identification both hands thumb, as the gesture with portable equipment
Input, to carry out the interaction with user.
In one example, electronic portable device is referred to by identification side outstanding four and is appeared in above electronic equipment
Or a thumb of side, to identify single hand grip posture.
In one example, it is contemplated that in the position of electronic portable device, Portable electronic is set two thumbs when both hands grip
The standby thumb for appearing in intelligent electronic device two sides by identification refers to two thumb fingers above root and screen, to identify both hands
Hold posture.
In one example, in the case where single hand grip, another hand to the touch action of grip hand include: finger by
Button and finger slider bar gesture, wherein in finger button gesture, to hold the finger of hand as button, another hand carries out it
It touches, in finger slider bar gesture, to hold the finger of hand as slider bar, another hand slides on or clicks.
In one example, the movement between the both hands thumb includes one or more of following: thumb touches, thumb
It is acted after finger wheel turns, thumb is touched by the movement of certain path, thumb respectively by certain path.It is dynamic by predefined paths about thumb
Make, such as thumb bending.
Three, the mobile phone biocular systems based on Softcam
We are by light reflecting device such as one piece of mirror surface (including eyeglass, prism, convex mirror or multi-panel lens combination etc.)
It is placed on above mobile phone camera, by the combination of camera and mirror, one can be provided for mobile phone inexpensive (only needs one
Block mirror) binocular vision system, two optical paths generated are cooperated by this block mirror surface and the existing camera of mobile phone, construct one
Virtual binocular camera (Softcam 1 and Softcam 2) restores the three-dimensional letter of object in captured space
Breath.
In typical stereo visual system, the same scene can generally be shot there are two camera.We will
Mirror and mobile phone have camera cooperation, using optical principle illustrated in Figure 14, build a virtual binocular camera,
To restore the three-dimensional information of its object within sweep of the eye.
Here mirror can be plane mirror, prism, convex lens etc., it is preferable to use prism, because prism has
The ability of inner full-reflection, so that compared to plane mirror is used, it is higher using the associated image quality of prism.
As shown in figure 14, the existing camera of mobile phone is by the reflection of mirror surface and mobile phone screen, constitutes two and virtual takes the photograph
As head, the light that object issues can respectively enter existing camera by two optical paths.
Wherein, optical path 1 indicates that one of optical path of object generation directly passes through mirror-reflection and enter camera；Optical path 2 indicates
The reflection that another road optical path that object generates first passes through mobile phone screen enters back into mirror surface, finally enters camera.Further, empty
Quasi- camera 1 is slightly above touch screen, this is the result of the primary event on prism mirror inclined-plane (optical path 1).Softcam 2 is by two
What secondary reflection light (optical path 2) generated, first reflection occurs on prism bottom or mobile phone screen, then reflexes to prism
Inclined-plane mirror surface.Two Softcams are parallel to mobile phone screen, together form stereo visual system.
This twice different light paths provides binocular view to mobile phone, constructs a biocular systems, so as to pass through meter
The depth information that the algorithm of calculation machine vision carries out object in space calculates, and obtains the three-dimensional information of object in space.
In one example, image of the intelligent electronic device based on acquisition, identify object, and based on identification as a result, come
It interacts.
For example, intelligent electronic device in the case where recognizing object is one-pen, estimates pen body and electronic curtain surface
Between angle, and based on the angle estimated, to control the interactive operation of intelligent electronic device；Or tracking pen tip is in desktop
The track of upper movement, and the track based on the pen tip identified, to control the interactive operation of intelligent electronic device, such as pen tip exists
On desktop when specific position click or move along certain track, indicate to carry out corresponding positions to the screen of intelligent electronic device
The click set is slided along corresponding track.
Intelligent electronic device can judge the identity of user based on the figure or clothing for recognizing user, and control and be somebody's turn to do
The interactive operation of user.Such as it is the equipment owner or manager that equipment, which recognizes the user, then carries out to the user more
Authorization；If recognizing user is non-equipment often user, opens less function or the equipment owner is notified.
In one example, intelligent electronic device also identifies the ambient enviroment of user, and judges locating occasion at this time, to adjust
Whole intelligent electronic device is correspondingly arranged.Such as user just in meeting room or cinema, equipment judges that user is in meeting ring at this time
Border, can automatically by device voice volume reduce or modulate it is mute, with interruption-free other people；If user is in driving environment, electronic equipment
Close all entertainment softwares etc..
According to an embodiment of the present invention, the virtual binocular camera can be captured using mobile phone bottom as a left side for starting point
Right transverse direction is 5 centimetres at least each, the image of the object at least 10 cm ranges longitudinally upward.
The embodiment of the present invention is directed to the existing front camera of mobile phone, combines with single prism, has devised and be suitable for
The light channel structure of mobile phone provides stereoscopic vision with single prism mirror and mobile phone screen, the cooperation of mobile phone front camera, and simultaneously
Front camera successfully " has been rotated " for shooting the view parallel with screen.
We have createed two virtual cameras, but obtained picture quality is not identical.It is virtual for entering
The optical path of camera 1 (optical path 1) is totally reflected on the inclined-plane of prism, therefore can obtain the figure being captured directly to camera
As the picture quality being not different.
And 2 situation of Softcam is more complicated.Prism bottom or mobile phone occur for the first reflection of this optical path
On screen, for there is a situation where in prism bottom, since the refractive index of prism glass is higher than air, this causes " anti-in complete
Penetrate " appearance of phenomenon, i.e. light is difficult to the contact surface (presence that must have air among contact surface) across prism and screen, but
It is completely reflected inside prism.Therefore, which, which generates, has brightness identical with the image in Softcam 1 and acutance
High quality graphic.For the latter, i.e., there is a situation where on mobile phone screen, first reflection occurs in mobile phone for first reflection
Light on screen can generate decaying since reflectivity is low.It will appear relatively in this image for causing front camera to capture
Dark region band, as shown in figure 15.By adjusting the relative position of prism and camera, we can make the width of blanking bar minimum
Change.Below, how we uses luminance compensation function removal dark portion before carrying out stereoscopic vision algorithm if introducing.
It according to an embodiment of the present invention, can for the binocular image of capture (while the two monocular RGB images obtained)
To execute following processing:
(1) it is corrected processing, wherein removing dark portion using luminance compensation function, obtains standardized monocular RGB figure
Picture；
(2) hand skin mask image is obtained using skin color segmentation method；
(3) two standardized monocular RGB images are based on, individual element calculates depth, obtains depth map；
(4) skin mask image and depth map are combined, the segmented image of hand region is obtained；
(5) segmented image based on obtained hand region carries out gesture identification.
In one example, correction process may include:
(1) color correction is carried out using pixel color equation Output=Input*R+L,
(2) wherein Output be output image in each pixel color, Input be input image in
The color of each pixel, R are reflection factor, are only dependent upon the physical property of screen surface, and L is the self-luminous of mobile phone.
In one example, be arranged the L be zero (such setting be it is reasonable, this is because our camera is non-
Very close to mobile phone surface, so the numerical value of L is almost nil under normal lighting conditions), and set by using the smart electronics
The standby image for acquiring white metope, fitting obtain parameter R.In more specific example, some white metopes are acquired with prototype
Image, and for each pixel using the parameter R in least square fitting model, and the black region in every frame image is removed with it
Domain.
About partitioning into skin technology, our task is in different lighting conditions and certain tone and saturation degree
In range, the background area obtained in image is eliminated as much as.Although extensive research has been carried out to skin detection in forefathers,
But existing solution (such as document 1, the research for the pixel classifications based on color) is not able to satisfy at us
Demand under special screne.Their purpose is detection all possible human skin color in same image, and we are only
The hand skin detection of a user need to be met.In addition, our skin detection module requires the Computationally efficient of real-time, interactive.
Based on considerations above, a set of skin detection algorithm is inventors herein proposed, this algorithm is divided into two modules a: mould
Block is simply by the threshold value of tone and saturation degree in image to divide skin area, another module is every a few frame dynamics
Calibrate these threshold values in ground.Particularly, it is introduced in the data set training such as document 5 that we are provided using document 1 and document 2,3,4
Convolutional neural networks identify user's hand skin pixels as semantic segmentation task.Utilize the skin of the embodiment of the present invention
Detection algorithm is suitable for real-time and efficiently carrying out one within the scope of different lighting conditions and certain tone and saturation degree
The hand skin of a user detects, and eliminates as much as the background area obtained in image.
The list of documents of reference is as follows:
Document 1:S.L.Phung, A.Bouzerdoum, and D.Chai.2005.Skin segmentation using
color pixel classification:analysis and comparison.IEEE Trans-actions on
Pattern Analysis and Machine Intelligence 27,1(Jan 2005),148–154.https://
doi.org/10.1109/TPAMI.2005.17。
Document 2:Tomasz Grzejszczak, Michal Kawulok, and Adam Galuszka.2016.Hand
landmarks detection and localization in color images.Multimedia Tools and
Applications 75,23(2016),16363–16387.https://doi.org/10.1007/s11042-015-2934-
5
Document 3:Michal Kawulok, Jolanta Kawulok, Jakub Nalepa, and Bogdan
Smolka.2014.Self-adaptive algorithm for segmenting skin regions.EURASIP
Journal on Advances in Signal Processing 2014,170(2014),1–22.https://doi.org/
10.1186/1687-6180-2014-170
Document 4:Jakub Nalepa and Michal Kawulok.2014.Fast and Accurate Hand
Shape Classification.In Beyond Databases,Architectures,and Struc-tures,
Stanislaw Kozielski,Dariusz Mrozek,Pawel Kasprowski,Bozena Malysiak-Mrozek,
and Daniel Kostrzewa(Eds.).Communications in Computer and Information
Science,Vol.424.Springer,364–373.https://doi.org/10.1007/978-3-319-06932-6_35
Document 5:ahundt, aurora95, unixnme, and PavlosMelissinos.2018.Keras-
tensorflow implementation of Fully Convolutional Networks for Se-mantic
Segmentation.https://github.com/aurora95/Keras-FCN.
The smart electronics portable equipment for being equipped with depth information acquisition and processing function can be more effective in conjunction with depth information
Ground identifies the hand motion and/or posture of user, such as identification grip electronic equipments grip position；Hold hand when electronic equipment
Finger movement；Click hand position used when screen；Click hand gesture used in screen；Hand extends in plane in electronic equipment
It clicks, sliding, scaling, selection；Finger of the hand above electronic equipment screen touches, pinches fist, hand, finger along particular path shifting
Dynamic movement.
In one example, smart electronics portable equipment identifies single hand grip posture, and identification another hand to grip
The touch action of hand, as the gesture input with portable equipment, to carry out the interaction with user.
In one example, smart electronics portable equipment refers to and appears in electronic equipment by identification side outstanding four
One thumb of top or side, to identify single hand grip posture.
In one example, portable equipment identifies the movement between both hands grip posture, and identification both hands thumb, as
With the gesture input of portable equipment, to carry out the interaction with user.
Movement between both hands thumb may include one or more of following: thumb touches；Thumb rotation；Thumb point
It An not certain path movement；Thumb is acted after touching by certain path.
Four, the intelligent electronic device of the hand of operation display is identified
According to one embodiment of present invention, a kind of intelligent electronic device is provided, (for example is touched comprising tangible surface
Screen or touch tablet, such as the touch tablet of notebook), have sensor, the sensor can capture it is above screen,
Hand images of user when using electronic equipment, wherein intelligent electronic device identification touches the state of the hand on screen, packet
It includes the concrete position of identification hand, identify related finger is one or more of which root finger, the direction of finger.
So-called " hand images of above screen, user when using electronic equipment " refer to conventional electronic devices in hand
When the screen of portion's face electronic equipment is operated, the video camera configured on electronic equipment is the hand that can not capture operation display
The image in portion, and the embodiment of the present invention configuration camera system (system here may include traditional front camera and
Additional device of configuration, such as plane mirror, prism etc.) can capture operation display hand image.
Here sensor can be common camera i.e. monocular cam binocular camera or infrared
Camera.
Here sensor can be in the side on tangible surface, in the following, above.
The concrete position of identification hand may include: identification finger tip, finger pulp, great thenar, articulations digitorum manus.Different finger positions
The same movement of same position is indicated to carry out corresponding operation to different objects.
As an example, finger tip touching indicates the operation to the fine target such as text, paintbrush, such as: using finger tip on text
Sliding, expression choose text in this block region to be operated；Paintbrush tool is used with finger tip, then is changed to current paintbrush brush small
Brush.
As an example, finger pulp touching indicates normal user's operation, such as the icon or menu option on commanding apparatus screen,
With select command, calls file, startup program or execute some other everyday tasks.
As an example, great thenar touching indicates the operation between application layer, such as: great thenar long-pressing on the screen indicates to return
Return homepage；Great thenar horizontally slips on the screen, indicates switching current application.
As an example, articulations digitorum manus touching indicates shot operation, such as: articulations digitorum manus double-clicks screen and represents interception current screen；Refer to
Joint is drawn a circle on the screen to be indicated to intercept the image in circle.
As an example, the concrete position for hand detects, in conjunction with screen capacitance signal and camera captured image signal,
It is completed by the traditional images processing technique such as Morphology observation.Specifically, it such as can carry out: be believed first according to screen capacitor as follows
Number position, the position of hand clicking point in the picture is determined by coordinate transform, so in the picture determine click on area (packet
Include the pocket above click location and click location), according to geometrical characteristic (depth information, the example of finger in the region
Such as inclination angle, joint position, arthrogryposis direction) and capacitance signal feature (such as contact area, the gradient of Contact Ellipse)
Equal multi-modal informations, determine hand for click concrete position.
As an example, identifying that related finger be which root finger may include identification thumb, index finger, middle finger, the third finger and small
Refer to.
As an example, different fingers can represent the different operation to same object, example to the same movement of same target
If middle finger clicks representation of file duplication, the third finger, which is clicked, to be indicated to paste；Index finger clicks wechat icon representation and opens wechat, middle indication
It hits sweeping in wechat icon representation opening wechat and sweeps application.
Such as different finger long-pressing homepages can indicate shortcut key, open respective application: for example index finger long-pressing can be with table
Show opening wechat, middle finger long-pressing can indicate to open Alipay etc..
In addition, different fingers can indicate that different tools, such as index finger indicate that paintbrush, middle finger indicate erasing rubber etc..
As an example, the angle of identification finger may include: the various angles for identifying finger relative to screen at 0 to 90 degree
Degree.These clicks to screen different angle, or the finger angular adjustment after clicking, can become information input.Such as: finger touching
After touching loudness/brightness regulation button, angle by adjusting finger relative to screen adjusts loudness/brightness size.
As an example, the angle of identification finger can carry out as follows: according to the position of screen capacitance signal, being become by coordinate
The position for determining it in the picture is changed, and then determines click on area (above click location and click location in the picture
One pocket), using the method for linear regression prediction, the point cloud of fingertip area in depth map is fitted, and then determines
Click the angle of finger.
The solution of the present invention can utilize one piece of mirror surface (including eyeglass, prism etc.), change the existing camera of mobile phone
Field range.The field range of camera is limited and fixed on conventional mobile phone, and such as front camera can only obtain right above mobile phone
Image information in 60*80 degree or so spatial dimension is not available hand information when mobile phone acquisition user uses mobile phone naturally,
It is many to be ignored to the valuable hand information of mobile phone interaction.This programme is by increasing inexpensive a mirror fitting and phase
The algorithm answered changes the field range of front camera, so that existing camera can capture user by the reflection of mirror surface
(when using mobile phone naturally) hand images information, the field range of cell-phone camera system has been widened significantly.
The solution of the present invention provides inexpensive (only needing an a face eyeglass) binocular vision system for mobile phone, passes through this
Two optical paths that block mirror surface and the existing camera cooperation of mobile phone generate, construct a virtual binocular camera, restore sky
Between middle object three-dimensional information.
The present invention also provides the information input sides based on touch screen based on the interaction gesture for holding hand, with conventional mobile phone
Formula is compared, and the gesture interaction based on grip hand can enrich the input mode of mobile phone, and some interactions are become more convenient.This
Outside, due to self perception of human body, user can easily position the physical feeling of oneself, therefore user can close without vision
The privileged site that grip hand is shone in the case where note interacts.This interactive mode of the embodiment of the present invention is more in line with user
Naturally use habit when mobile phone is used, does not need purposely lift using hand, makes unnatural gesture in space.It is real
The user experience experiment on border shows that these interaction design modes can be received by user well, while having entertaining
Property, learnability and convenience.
Above using mobile phone as the example of smart electronics portable equipment, but it is also possible to other intelligence for capableing of hand gripping
Electronic portable device product, such as intelligent vehicle-carried electronic equipment, Intelligent flat computer etc..
Various embodiments of the present invention are described above, above description is exemplary, and non-exclusive, and
It is not limited to disclosed each embodiment.Without departing from the scope and spirit of illustrated each embodiment, for this skill
Many modifications and changes are obvious for the those of ordinary skill in art field.Therefore, protection scope of the present invention is answered
This is subject to the protection scope in claims.
Claims (10)
1. a kind of intelligent electronic device, is deployed with camera, the camera can capture the grip hand of user's holding apparatus
Image,
The image of grip hand of the intelligent electronic device based on user obtained, the gesture of identification grip hand,
The gesture of grip hand based on identification, intelligent electronic device execute corresponding control operation.
2. intelligent electronic device according to claim 1, the gesture of the identification grip hand includes holding around tracking electronic equipment
The position of finger is held, and using the movement for lifting, moving or tapping intelligent electronic device of different grip fingers as interaction
Information input.
3. intelligent electronic device according to claim 1, the grip gesture model includes that hand-held intelligent electronic equipment is taken pictures
Gesture is referred to as gesture of taking pictures,
When intelligent electronic device, which recognizes, takes pictures gesture, intelligent electronic device starts the application taken pictures automatically, and automatically snaps
One photo.
4. intelligent electronic device according to claim 1, additionally it is possible to which operation comes:
Identify that user is held with which hand；
Based on recognition result, graphic user interface layout is adjusted, enable user grip hand finger compared to
Target component is more easily clicked before adjustment.
5. intelligent electronic device according to claim 1, camera is at the edge of intelligent electronic device.
6. intelligent electronic device according to claim 1, camera is the fish-eye camera below screen.
7. intelligent electronic device according to claim 1 further includes the infrared illumination being used in combination with camera and infrared filtering
Piece, to increase signal-to-noise ratio.
8. intelligent electronic device according to claim 1, camera is depth camera.
9. intelligent electronic device according to claim 1, further include the screen arrangement for favouring intelligent electronic device optics it is anti-
Injection device,
The light reflection unit can light of the reflected parallel in electronic equipment screen surface capture it by camera, thus with
When electronic equipment is held at family, camera can capture the image of hand of above the screen, user when holding electronic equipment；
Hand images of the intelligent electronic device based on user obtained identify the hand motion and/or posture of user at this time, make
For the input information of user, and interacted with user.
10. a kind of man-machine interaction method of intelligent electronic device, the intelligent electronic device is deployed with camera, the camera
The image of the grip hand of user's holding apparatus can be captured,
The man-machine interaction method includes:
The image of grip hand based on user obtained, the gesture of identification grip hand,
The gesture of grip hand based on identification, electronic equipment execute corresponding control operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910154756.4A CN109963034A (en) | 2019-03-01 | 2019-03-01 | The capture of intelligent electronic device gesture and identification technology based on grip hand detection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910154756.4A CN109963034A (en) | 2019-03-01 | 2019-03-01 | The capture of intelligent electronic device gesture and identification technology based on grip hand detection |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109963034A true CN109963034A (en) | 2019-07-02 |
Family
ID=67023913
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910154756.4A Pending CN109963034A (en) | 2019-03-01 | 2019-03-01 | The capture of intelligent electronic device gesture and identification technology based on grip hand detection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109963034A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN201035553Y (en) * | 2007-04-10 | 2008-03-12 | 北京汇冠新技术有限公司 | Light path structure of touch panel using camera and reflector |
| CN104951052A (en) * | 2014-03-24 | 2015-09-30 | 联想(北京)有限公司 | Information processing method and electronic equipment |
| KR20170037123A (en) * | 2015-09-25 | 2017-04-04 | 엘지전자 주식회사 | Mobile terminal and method for controlling the same |
| CN206323415U (en) * | 2016-10-25 | 2017-07-11 | 深圳奥比中光科技有限公司 | A kind of electronic equipment with rotatable camera |
| CN108268195A (en) * | 2018-02-28 | 2018-07-10 | 努比亚技术有限公司 | One-handed performance display methods, mobile terminal and computer readable storage medium |
| CN208094617U (en) * | 2018-04-23 | 2018-11-13 | 珠海格力电器股份有限公司 | Electronic equipment |
-
2019
- 2019-03-01 CN CN201910154756.4A patent/CN109963034A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN201035553Y (en) * | 2007-04-10 | 2008-03-12 | 北京汇冠新技术有限公司 | Light path structure of touch panel using camera and reflector |
| CN104951052A (en) * | 2014-03-24 | 2015-09-30 | 联想(北京)有限公司 | Information processing method and electronic equipment |
| KR20170037123A (en) * | 2015-09-25 | 2017-04-04 | 엘지전자 주식회사 | Mobile terminal and method for controlling the same |
| CN206323415U (en) * | 2016-10-25 | 2017-07-11 | 深圳奥比中光科技有限公司 | A kind of electronic equipment with rotatable camera |
| CN108268195A (en) * | 2018-02-28 | 2018-07-10 | 努比亚技术有限公司 | One-handed performance display methods, mobile terminal and computer readable storage medium |
| CN208094617U (en) * | 2018-04-23 | 2018-11-13 | 珠海格力电器股份有限公司 | Electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11546505B2 (en) | Touchless photo capture in response to detected hand gestures | |
| US11531402B1 (en) | Bimanual gestures for controlling virtual and graphical elements | |
| US20220326781A1 (en) | Bimanual interactions between mapped hand regions for controlling virtual and graphical elements | |
| US11861070B2 (en) | Hand gestures for animating and controlling virtual and graphical elements | |
| Nivash et al. | Implementation and Analysis of AI‐Based Gesticulation Control for Impaired People | |
| WO2021023059A1 (en) | Photographing method and mobile terminal | |
| US9310891B2 (en) | Method and system enabling natural user interface gestures with user wearable glasses | |
| CN105027033B (en) | Method, device and computer-readable media for selecting Augmented Reality object | |
| US8854433B1 (en) | Method and system enabling natural user interface gestures with an electronic system | |
| US10477090B2 (en) | Wearable device, control method and non-transitory storage medium | |
| CN110073316A (en) | Interaction virtual objects in mixed reality environment | |
| US20200301513A1 (en) | Methods for two-stage hand gesture input | |
| KR20150143612A (en) | Near-plane segmentation using pulsed light source | |
| MX2009000305A (en) | Virtual controller for visual displays. | |
| CN108829242A (en) | Intelligent terminal and its non-touch operating method | |
| CN104813258A (en) | Data input device | |
| CN109947243A (en) | Based on the capture of intelligent electronic device gesture and identification technology for touching hand detection | |
| CN106201173A (en) | The interaction control method of a kind of user's interactive icons based on projection and system | |
| KR101256046B1 (en) | Method and system for body tracking for spatial gesture recognition | |
| JP5569973B2 (en) | Information terminal device, method and program | |
| CN109960406A (en) | Based on the intelligent electronic device gesture capture acted between both hands finger and identification technology | |
| KR20140132906A (en) | Device and method for mobile tooltip using eye detecting | |
| KR100749033B1 (en) | A method for manipulating a terminal using user's glint, and an apparatus | |
| CN109993059A (en) | Binocular vision and object recognition technique on intelligent electronic device based on single camera | |
| CN110442242B (en) | Intelligent mirror system based on binocular space gesture interaction and control method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20190702 |