WO2017009707A1 - Apparatus and method for hybrid type of input of buttons/keys and "finger writing" and low profile/variable geometry hand-based controller - Google Patents

Apparatus and method for hybrid type of input of buttons/keys and "finger writing" and low profile/variable geometry hand-based controller Download PDF

Info

Publication number
WO2017009707A1
WO2017009707A1 PCT/IB2016/000998 IB2016000998W WO2017009707A1 WO 2017009707 A1 WO2017009707 A1 WO 2017009707A1 IB 2016000998 W IB2016000998 W IB 2016000998W WO 2017009707 A1 WO2017009707 A1 WO 2017009707A1
Authority
WO
WIPO (PCT)
Prior art keywords
finger
user
hand
input
controller
Prior art date
Application number
PCT/IB2016/000998
Other languages
French (fr)
Inventor
Quan Xiao
Original Assignee
Quan Xiao
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Quan Xiao filed Critical Quan Xiao
Priority to PCT/IB2016/000998 priority Critical patent/WO2017009707A1/en
Priority to EP16823943.2A priority patent/EP3323036A4/en
Priority to CN201680053129.6A priority patent/CN108027705A/en
Publication of WO2017009707A1 publication Critical patent/WO2017009707A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0354—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
    • G06F3/03547—Touch pads, in which fingers can move on a surface
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/20—Input arrangements for video game devices
    • A63F13/21—Input arrangements for video game devices characterised by their sensors, purposes or types
    • A63F13/214—Input arrangements for video game devices characterised by their sensors, purposes or types for locating contacts on a surface, e.g. floor mats or touch pads
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/40—Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
    • A63F13/42—Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16—Constructional details or arrangements
    • G06F1/1613—Constructional details or arrangements for portable computers
    • G06F1/163—Wearable computers, e.g. on a belt
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/02—Input arrangements using manually operated switches, e.g. using keyboards or dials
    • G06F3/0202—Constructional details or processes of manufacture of the input device
    • G06F3/0216—Arrangements for ergonomically adjusting the disposition of keys of a keyboard
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0354—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
    • G06F3/03543—Mice or pucks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05G—CONTROL DEVICES OR SYSTEMS INSOFAR AS CHARACTERISED BY MECHANICAL FEATURES ONLY
    • G05G9/00—Manually-actuated control mechanisms provided with one single controlling member co-operating with two or more controlled members, e.g. selectively, simultaneously
    • G05G9/02—Manually-actuated control mechanisms provided with one single controlling member co-operating with two or more controlled members, e.g. selectively, simultaneously the controlling member being movable in different independent ways, movement in each individual way actuating one controlled member only
    • G05G9/04—Manually-actuated control mechanisms provided with one single controlling member co-operating with two or more controlled members, e.g. selectively, simultaneously the controlling member being movable in different independent ways, movement in each individual way actuating one controlled member only in which movement in two or more ways can occur simultaneously
    • G05G9/047—Manually-actuated control mechanisms provided with one single controlling member co-operating with two or more controlled members, e.g. selectively, simultaneously the controlling member being movable in different independent ways, movement in each individual way actuating one controlled member only in which movement in two or more ways can occur simultaneously the controlling member being movable by hand about orthogonal axes, e.g. joysticks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/033—Indexing scheme relating to G06F3/033
    • G06F2203/0331—Finger worn pointing device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures

Definitions

  • the present invention relates to hand based "hybrid" input device (human computer interface), more specifically related to methods and apparatus to allow compact, single hand operable symbol and character input methods that do not rely on vision and utilizing skills user already have.
  • a new kind of dynamic/shape-transformable human-machine interaction means is also discussed here, more precisely relating to Variable Geometry hand-based controller, for providing on-demand, (Ergonometric and) change-able (transformable) interface for user's hand to interact with, thus provide more flexible/versatile way of interaction and possibly simulating (including the user experience) a variety of (hand-based) controllers/devices, so that user can have more than one way of interaction without need to switch to other controllers.
  • controllers such as joystick, mouse, game pads, touch pads, keyboard, data gloves, wii motes and etc. are commonly used to (and come in pair) with one or more machine being controlled.
  • controller usually have controls on surface of the controller such as buttons, touch pads, joysticks and haptic feedbacks , and the layout usually customized to the specific interaction between user and machines, for example, the frequent used controls are made bigger and easy to reach by the fingers.
  • a suggested solution provided in this invention by using a unique variable (change-able) geometry hand operated apparatus that can change/transform between multiple geometry shapes/forms, with at least one of these form is functional for hand/finger manipulation (of the controls on the device/controller), thus to provide variable interfaces to user's hand and allow "on-demand”/dynamic changing of the interfaces, thus allow much more flexible manipulation and integration of multiple controls/devices easily without need to physically change the controller in user's hands, and provide user with unique experiences while still take advantage of the manipulation skills for the controls they already have.
  • a low-profile joystick like control from using touch sensitive area is also discussed here. (Optionally) It can be used in the Variable Geometry hand- based controller (as controls for user to interact with).
  • Finger writing A pattern (or a sequence of touch events/strokes) on a surface touched/draw by the tip of a finger (such as index finger, middle finger or top part of the thumb), ⁇ basically characters, symbols, or navigation (such as swipe, rotate) "marks" written on the surface by using finger tip (or top part of the thumb).
  • the resulting pattern could be similar to that of a handwriting, however finger writing does not need to have a pen or stylus involved and only need one finger to draw the pattern, (could also mean the act of drawing such pattern)
  • the corresponding hand position for this "normal/comfortable" hand/finger writing position when using index finger or middle finger is palm facing downwards, or facing sideway that is not considered towards user himself/herself, while for thumb writing in some situations the hand position is for the palm to be partially upwards or partially towards user, and not quiet desirable for facing away from user).
  • Game controller for user interacting with computer system/game console usually have multiple "axis" of control such as with joysticks and buttons/keys and/or touch areas on it,
  • the "Layout" of the controls normally allows multiple controls being manipulated/operated simultaneously by different fingers, and user have a "normal” holding pose/gesture (such as 2 hands on the handle with thumbs on the joysticks/keypads) are considered the “standard” or "normal” (by design) posture/gesture for playing the game".
  • the new touch pad finger writing input method in this invention is also based on such standard/normal gesture(pose of hand/fingers), and allow index/middle finger to do text input without (significantly/substantially) interrupting other finger's operation.
  • finger/thumb operate/manipulated by user's finger/thumb, such as but not limit to button(s), key(s), joystick(s) (especially gamepad like), touch sensitive areas such as but not limited to (touch pad/touch screen and etc).
  • Partial (tooth) gears - gears (including bevel gears) or pair of gears with only a part of the whole arc having tooth, other part are either smooth (and not driving the movement of the counterpart/meshing gear) or does not exists.
  • system/appliances which the device/controller connected or controlling) that do not require the help from (user's) vision or voice includes:
  • touch sensitive means such as but not limited to touch pad, touch screen and etc.
  • touch sensitive means such as but not limited to touch pad, touch screen and etc.
  • touch sensitive means that is on the surface of or articulated to said handheld/wearable device at an (ergonometric) location that have a surface facing direction for "normal'Vcomfortable hand/finger writing, and can be easily reached by user's finger tip in a "normal'Vcomfortable hand/finger writing position;
  • the size of the touch sensitive area is no less than 6mm x 6mm, and maybe not need to be bigger than the size to accommodate a normal finger writing character/symbol);
  • said touch area might have boundaries/(tactile) hints to indicate the effective touch sensitive area for finger writing input, such as but not limited to: protrusion/recess on the surface or raised/ depressed areas, edges, bumps and etc , it also could be surface (material)
  • Sensing user's "touch information" (“finger writing") on the touch sensitive area such as but not limited to touch location/coordinates, events, patterns, force/magnitude (if the pad can provide) and etc.
  • the recognition result might be combined with other input (if any) from user such as key/button user holding/pressed or gesture user is making (at the same time user performing the finger writing) to form the result of the output
  • the data/events/information from said touch pad/area is transmitted to said finger writing recognition module in a "real-time or near-real-time” fashion such as but not limited to events/information "packets" about the location/coordinate of the current touch point and timestamp, or the "segment” of the current (continues) curve, or it could be the pattern when touch system determines that the whole character is inputted;
  • user can be prompt (visually or audibly) for candidates of recognition, or auto-completion;
  • the method further includes combining information from buttons/keys/touch sensitive areas operable by finger(s) other than the one used for handwriting, so that user can use the combination of keys (together with the handwriting) for more complicated and convenient input, such keys/buttons can provide functions such as CAPs, Shift, control, Fn, symbols, space, return and other not “visible” characters/functions, or allow user to quickly select from candidates/auto completion.
  • keys/buttons can provide functions such as CAPs, Shift, control, Fn, symbols, space, return and other not “visible” characters/functions, or allow user to quickly select from candidates/auto completion.
  • the output character/symbol might be in a interface similar or simulating to that of a keyboard interface
  • a pair of in the controller is a pair for both hands
  • a wearable interactive device/controller that allows multiple fingers to provide input (to the computer system it connects to) simultaneously
  • a game controller without interfering existing functionalities/finger operations/game play (such as the ability to operate the joystick with the thumb, and other buttons)
  • touch sensitive means such as but not limited to touch pad, touch screen and etc.
  • handheld/wearable device and the touch area have a surface facing direction for
  • Such "convenient reachable" location of touch area /means is on (the surface) or articulated to said input device (such as but not limited to the game controller) that can be reached by index finger or middle finger and perform "finger writing" while maintaining a normal operation gesture for other controls/buttons on the input device/controller (such as a game controller) - in another words, the location of the touch area is within the range of the index and middle finger when user hold a "normal operation status" for normal
  • said touch area pad might be modular and can be added/removed from the controller.
  • said touch area pad might be
  • 3D controller which provides positional/movement input (to the computer system it connects to) without interfering existing functionalities/ operations/navigation includes:
  • touch sensitive means such as but not limited to touch pad, touch screen and etc.
  • Such "convenient reachable" location of touch area /means is on (the surface) or articulated to said input device (such as but not limited to mouse, 3D mouse, 3D controller) that can be reached by index finger or middle finger and perform "finger writing" while maintaining a normal operation gesture the device; for example the touch area could be on the "left” or “middle” button of the mouse, or it could be in the area that is about the location of "middle button” or further up, so that left-handers can also operate such touch pad/area with ease;
  • said input device such as but not limited to mouse, 3D mouse, 3D controller
  • Performing "finger writing" recognition (by recognition modules on board the controller or transmitted the information/data/events of finger writing from the controller to the (host) computer system connected), it is desirable such recognition might take consideration of (if any) simultaneous input by other fingers - such as key/button press from additional keys on the controller such as but not limited to additional keys/function keys on side of the mouse/3D mouse or 3D controller which can be operated by user's other finger at the same time) or gestures detected by the controller/input system to form the end output result (for example if user hold down Ctrl or Fn key, the output result is like user hold down these keys and type on the keyboard);
  • additional keys on the controller such as but not limited to additional keys/function keys on side of the mouse/3D mouse or 3D controller which can be operated by user's other finger at the same time
  • gestures detected by the controller/input system to form the end output result (for example if user hold down Ctrl or Fn key, the output result is like user hold down these keys and type on
  • a Variable Geometry hand-based control apparatus comprised of:
  • At least 2 articulated or touched members that can rotate/move/displacement against each other and at least one of the members can fit into the operation range of a user's hand/fingers (such as but not limited to: the size fits into user's palm), and one or more controlled actuator/powered driving mechanism on the apparatus and operationally connected/coupled to at least one(or 2) members that is able to rotate/move/displacement at least one "active/movable" member rendering the geometry form/shape change from to an other following control from a controller for form/shape control (which desirably could , either coordinates with or controlled by or can processing request from, appropriate user interface(s) so that the current form of the apparatus is consistent with the current (intended) user interface for proper interaction requirements/preferences which could be different for different apps/software that needs interaction/input from user);
  • a controller for form/shape control which desirably could , either coordinates with or controlled by or can processing request from, appropriate user interface(s) so that the
  • the geometry form/shapes of said hand-based control apparatus allows user to use hand/finger to operate the controls on the "surface for hand/finger control”—such as but not limited to: performing control of machine connected to the apparatus, performing input/interaction such as but not limit to touch input, pressing keys/buttons, manipulating joystick and etc.;
  • the apparatus allows user to perform control/interaction with a computer implemented system (that expects/requires user interaction) in different manner/form without the need to change to another controller, and might also be able to simulate the physical experience of different kind of controls.
  • (movable) members that can move/rotate against each other, the members might be moved in a different time sequence and not always move at the same time.
  • said movement of members has a sequence of movement with least 2 timing/phase(or step) for movement of members or transformation (members move at different timing/phase of a sequence of movement, similar to what is depicted in Fig.3 a );
  • At least 2 different timing/step/phase of operation/movement in said sequence of are driven by a common actuator (such as but not limited to servo) using a drive train (such as but not limited to shaft and transmission system such as partial (tooth) gears) that is capable of turning actuator output at different timing/phase (such as but limited to: rotational angle, travel distance and etc) into movement of different members (at different time) (for example but not limited to the drive train system depicted in Fig l.d and Fig 3).
  • a common actuator such as but not limited to servo
  • a drive train such as but not limited to shaft and transmission system such as partial (tooth) gears
  • At least 2 surface for hand/finger control can be formed at an angle to each other and both within hand/finger reach at the same time so that user can use different finger/thumb to operate different controls on these surfaces at the same time.
  • one of these form is a low- profile/non-blocking/minimum-interference form (such as but not limited to the form depicted in Fig.1 (c ) ) to allow user to use hand/fingers for other purpose/operations (such as but not limit to gesture input, pressing other buttons or operate other devices without the need for removing/releasing this apparatus from hand), and at least one form/shape is a "functional" form where the controls on the "surface for hand/finger control" can be reached conveniently and manipulated.
  • the mechanism can rendering (at least) 2 planes (one for thumb/joystick) and one for index/other fingers to change (such as rotate, like 90 deg ) from collapsed position to expanded position, and might cause the 2 planes interlock to form a stable controller.
  • the at least one of the folding axis of a moveable member is (substantially) parallel to that of the average folding axis (vector) of the fingers of a user's hand excepting the thumb.
  • a controller have finger/hand- operable controls on the surface for the other hand of user that can use together with this Variable Geometry hand-based control apparatus to form a full controller;
  • the 2 controller can be attached/coupled together when the Variable Geometry hand-based control apparatus is in (some) stable form(s) such as but not limited to when it is expanded to a "full grip” mode, so that for example user can use the attached/coupled controllers with both hands as “one piece", maybe (but not limited to) like a standard gamepad.
  • the finger- operable controls on the surface of the controller and the "layout" is similar to the
  • Secondary side (usually “left side") of a game pad, maybe depends on user is right-handed or left-handed (in which case it is the right side of a game pad, and Variable Geometry hand- based control apparatus to form a full controller is in user's left hand);
  • a control stick or joy stick having at least one driving mechanism (such as but not limited to an actuator connected to a shape /form control system) in addition to normal control stick/joystick components, and at least one articulated/touched "active/movable” member(s) having hand/finger-operable control(s) on it (such as but not limited to touch sensitive area such as touch pad/screen, buttons, knobs and etc.) that can be driven by said driving mechanism to move/rotate (such as but not limited to "rotate up” or "pop up”) to a position/location that is easily reachable by user's fingers/hand, desirably on demand or dynamically following the control of the shape/form control system, so that user can operate the controls on said "active/movable” members simultaneously when manipulating the control/joy stick and/or the "original" controls on the stick.
  • driving mechanism such as but not limited to an actuator connected to a shape /form control system
  • the controls on the "active/movable" members including a touch pad or touch screen which operationally connect to a touch and finger/hand writing recognitions system so that user can use the touch pad/screen to perform text input via finger/hand writing recognition system while at the same time operate the control/joy stick as well as other controls on it.
  • a game controller such as a gamepad or Wii mote like controller
  • remote controller having at least one driving mechanism (such as but not limited to an actuator connected to a shape /form control system) in addition to normal control stick/joystick components, and at least one articulated/touched
  • active/movable member(s) having hand/finger-operable control(s) on it (such as but not limited to touch sensitive area such as touch pad/screen, buttons, knobs and etc.) that can be driven by said driving mechanism to move/rotate (such as but not limited to "rotate up” or “pop up") to a position/location that is easily reachable by user's fingers/hand, desirably on demand or dynamically following the control of the shape/control system, so that user can operate the controls on said "active/movable” members simultaneously when manipulating the game controller/remote and the "original" controls" on said game controller/remote.
  • hand/finger-operable control(s) on it such as but not limited to touch sensitive area such as touch pad/screen, buttons, knobs and etc.
  • the controls on the "active/movable" members including a touch pad or touch screen which operationally connect to a touch and finger/hand writing recognitions system so that user can use the touch pad/screen to perform text input via finger/hand writing recognition system while at the same time operate the game controller/remote as well as other controls on it.
  • the apparatus is wearable (or: can accommodate user's hand and/or arm and move together) such as but not limit to in a shape of glove/half glove or have strap on it (such as on member 112 in Fig 1) so that it can be secured to the user's hand/palm/wrist/arm (the limb/part of body considered wearing devices and can perform control), or have means for attaching to/coupling with a glove/wearable device and move together with hand or arm.
  • wearable or: can accommodate user's hand and/or arm and move together
  • the apparatus is wearable (or: can accommodate user's hand and/or arm and move together) such as but not limit to in a shape of glove/half glove or have strap on it (such as on member 112 in Fig 1) so that it can be secured to the user's hand/palm/wrist/arm (the limb/part of body considered wearing devices and can perform control), or have means for attaching to/coupling with a glove/wearable device and move together with hand or arm.
  • the apparatus can change/transform between multiple (stable) geometry forms simulating the size of the tool and desirably also (at least in some of the forms) the control location relative to hand (similar to that of the original tool/device being simulated), for example (but not limited to : ) , with the movement of at least one driving mechanism (such as but not limited to the rotation or travel of a servo) that drive(s) at least one member to move/rotate/displace so that the geometry from of the device changing from low profile "storage'Vminimum blocking form, to a form simulating small handler (thin handle) such as a dagger (with slightly bigger profile, this can be done in a way for example but not limited to: in Fig.1 control the servo 105 to rotate a small degree such as 15 or 20 degree comparing to the fully expanded degree in this
  • the motion/position tracking means desirably is located on the "palm" side of the hand such but not limit to on or close to member 112 or 113 (in overreached form);
  • variable geometry controller to simulating a physical user experience of a joystick's including: using touch sensitive area (such as but not limited to a touch pad or touch screen) together with a elastic film on top of the (primary) touch area so that user can feel the force from their fingers when the control is not centered and the direction and aptitude of the force indicate (by tactile sense) the "drift" or "distance” from the center.
  • touch sensitive area such as but not limited to a touch pad or touch screen
  • a simulated joystick apparatus comprised of: a touch area (pad, screen) input, a thin elastic film which can be added on top of the touch area (such as but not limited to by means of attaching/coupling) , and it is desirable optionally the elastic film have protrusion/button/knob at or close to the center of the control/touch area, or tactile markings/patterns such as on the surface facing user or some form of tactile pattern so that user can feel the difference when finger touching different areas (which help him/her to use tactile sense only to locate or find center of the control without having to look at it; So that when user press the elastic film (by finger or by pressing central button/knob if available) a small part of the corresponding film in turn touches the touch sensitive area, and the touch sensitive areas (such as but not limit to touch pad, touch screen) can detect the "touch point(s)" x, y coordinates which can be translated into joystick x,y input by comparing the x, y coordinates of the
  • Teflon or other with resistance reduction coating can be applied to the other side of the film which facing the touch area surface so that to make it easy to move(drag) around the touch area (such as touch pad/screen) with user's finger pressed the elastic film or pressed the central button/knob.
  • the elastic film (such as but not limited to rubber) can be placed in a frame, such film can have a
  • docking/coupling mechanism such as but not limited to mechanical coupling means such as screw, lip and etc. or magnetic coupling mechanism such as magnets or other methods such as sticky surfaces (heat sensitive glue and etc) on the frame of the film and corresponding counterparts on/around the touch sensitive area) that fix/securing the film on top of (and cover a least a part of) the touch sensitive area(such as pad/touch screen) (of a controller) to form a (low profile) joystick, while it can also be removed when necessary so that user can directly operate the touch sensitive area;
  • mechanical coupling means such as screw, lip and etc.
  • magnetic coupling mechanism such as magnets
  • other methods such as sticky surfaces (heat sensitive glue and etc) on the frame of the film and corresponding counterparts on/around the touch sensitive area) that fix/securing the film on top of (and cover a least a part of) the touch sensitive area(such as pad/touch screen) (of a controller) to form a (low profile) joystick, while it can also be removed when necessary so
  • a knob/protrusion(s)/button (which maybe in a shape and size similar to that of a gamepad joystick's header part that touch's user's ginger/thumb) can be placed on the "center (area)" of said elastic film so it feels like a joystick to the finger and can be operated in a similar way.
  • the touch sensate area( or size) of the pad can also be (software) adjusted (such as but not limited to adjusting the "scale" when converting from touch pad x,y coordinates to joy stick x, y input as mentioned above) so that to accommodate different needs of user, generally speaking a larger area allows more accurate movement/aiming while a smaller area allows more nimble/fast movement/turning input.
  • the film and the touch area having a similar shape and size, in implementation they do not have to be the same (or very similar) shape, for example but not limited to: the shape of the film can be round and the shape of the touch sensitive area (such as pad or touch screen) could be square/rectangular (maybe with round edges), and generally the size of the touch sensitive area is about the same or larger than that of the film. It is possible that the film only cover a part of the touch sensitive area (especially in the situation of a touch screen where only a part of the screen is "allocated" for the purpose of "joystick like input simulation".
  • a round to square/rectangular "adapting" frame can be used to secure a round shaped film to a rectangular touch sensitive area (using coupling means such as but not limit to mechanical, attach/adhesive (heat sensitive adhesive), magnetic (using magnets) and etc.)
  • This a low profile joystick control can be used in conjunction or as a control in the Variable Geometry hand-based control apparatus as described above.
  • the control used on at least one of the Surface for hand/finger control (such as but not limited to 110 in Fig.1) is a low profile joystick control as described above.
  • Fig 1 shows one kind of Variable Geometry Controller that can be combined with a hand wearable device (not shown).
  • 112 can attach to a wearable device/harness and placed in user's palms and it could be in a form that curved along the general/average direction of "palmer crease(s)" or "folding seam" of a user's hand (102) that allow the upper part and lower part of user's hand to fold/ relative rotate.
  • [0089] 115 is a second "surface for hand/finger control" with controls such as touch pad/joystick 110 and buttons 111 on it and can be rotated between a operation/functional position/configuration as shown in (b) and a low-profile/ non-blocking position in (a) or (c). 113 can also rotate between an operational form/configuration such as (a) or (b) to a low- profile/non-blocking position/shape/form like in (c). The changes can be driven by actuator / driving mechanisms such as 105 (a servo) and related drive trains (such as but not limited to shaft and transmission system such as partial (tooth) gears) depicted in the figure.
  • actuator / driving mechanisms such as 105 (a servo) and related drive trains (such as but not limited to shaft and transmission system such as partial (tooth) gears) depicted in the figure.
  • Fig.1 (c) shows the "folding/collapsed" form/shape of the control which form a low profile, non-blocking shape that allow user to perform other operations/input with hand/fingers such as using gesture or press buttons/touch pad of other device.
  • Fig 1 (d) shows an example of driving mechanism where actuator (such as a servo) 105 is driving a (transmission) shaft 121 trough bearings 122 that is fixed on member 112, the shaft rotates around axis 101 and drive partial (tooth) gears (pair/train) 123 that cause member 113 which fixed to 105 to rotate against 112 (which usually have no relative movements against user's palm).
  • actuator such as a servo
  • 105 is driving a (transmission) shaft 121 trough bearings 122 that is fixed on member 112
  • the shaft rotates around axis 101 and drive partial (tooth) gears (pair/train) 123 that cause member 113 which fixed to 105 to rotate against 112 (which usually have no relative movements against user's palm).
  • the partial (tooth) bevel gear pair 125 drives shaft 126 through bearings 127 fixed on 113 and cause it to rotate along axis 103 and through supporting structures such as but not limited to 128 which fixed on member 115 to cause it to rotate (raise up and down) and change between a folded form and a expanded form where user's thumb can operate it comfortably like Fig 1. (b).
  • the partial (tooth) gear (some part have tooth and some part are smooth and not “driving" the counter part gear, such as but not limited to those depicted in Fig 3 (b) ) and the configuration of it allows 112 and 115 to be turned at different time/phase of the turning of 105 and usually not actuated at the same time (while there could be overlapping).
  • Fig.2 shows a low profile joystick like control can be formed by adding a film of elastic on top of a touch area 201, and a frame of the film (and related coupling means in 205 and 202) can be used to secure the film on top of the touch area and desirable removable/exchangeable.
  • Fig 2 (a) shows different "style" of elastic films 203 and 203' (such as those with different tactile patterns as shown in 203', or some film have a central button/knob 204 like the top/head part of a joystick) can be placed in a frame 205 or 205' .
  • Fig 2(b) shows a rotation/folding way of securing the film.
  • controller such as remote control for smart TV/smart appliances
  • 301 has a touch surface/area for finger writing input such as on top left area (shown as 303) operable for both index finger and thumb, that can be reached by index or middle finger or thumb for finger writing input for character/symbol.
  • buttons 302 which can be defined (such as function keys such as but not limited to Ctrl, shift,alt, Fn) and use by other finger(s).
  • FIG 4 shows A mouse (or 2D or 3D position/motion device) 401, can have a touch surface/area for finger writing input such as on left button (shown as 403 A), or in the middle section shown as 403B "above"(or to the upper side of) middle button/roller 402, that can be reached by index or middle finger for finger writing input for haracter/symbol.
  • a touch surface/area for finger writing input such as on left button (shown as 403 A), or in the middle section shown as 403B "above"(or to the upper side of) middle button/roller 402, that can be reached by index or middle finger for finger writing input for haracter/symbol.
  • buttons 405 which can be defined (such as function keys such as but not limited to Ctrl, shift,alt, Fn).
  • Fig 5 shows a wearable controller (such as a glove like game controller) could have (foldable/modular) controls and the touch area for "finger writing input" on 2 planes (one for index finger write input, one for thumb joystick like input) that at angle to each other (such as but not limit to perpendicular), (1) is the touch area for index finger for finger writing input and (2) is the touch area or joystick/button(s) for thumb to control direction, just like on a normal game pad/controller. (1) and (2) might be foldable/collapsible when they are not in use, so that they wont obstacle/hinder hand/fingers performing other activity/gestures.
  • FIG. 6 shows A typical game controller 601, which could have one or more joystick 602 which can be operated by thumb, and could have a touch pad area 605 for directional input. It can also have a touch surface/area for finger writing input such as on top left area (shown as 603B), or it could be pop up, foldable or "add-on" (modular) type as shown in 603A , both positions can be reached by index or middle finger for finger writing input for character/symbol.
  • a typical game controller 601 which could have one or more joystick 602 which can be operated by thumb, and could have a touch pad area 605 for directional input. It can also have a touch surface/area for finger writing input such as on top left area (shown as 603B), or it could be pop up, foldable or "add-on” (modular) type as shown in 603A , both positions can be reached by index or middle finger for finger writing input for character/symbol.
  • buttons which can be defined (such as function keys such as but not limited to Ctrl, shift,alt, Fn).
  • Fig 7 shows a single hand type game controller 701 which include position tracking/motion sensing, it can also have a touch surface/area (such as a tich pad) 702 for finger/thumb writing input, or similar to Fig.6 it can also have be pop up, foldable or "add-on"(modular) type touch pads , and optionally side buttons which can be defined (such as function keys such as but not limited to Ctrl, shift,alt, Fn).
  • a touch surface/area such as a tich pad
  • Fig.6 it can also have be pop up, foldable or "add-on"(modular) type touch pads , and optionally side buttons which can be defined (such as function keys such as but not limited to Ctrl, shift,alt, Fn).
  • Fig. 8 shows different timing/phase (such as in Fig.8 (a) ) of the partial (tooth) gear (as shown in Fig 8 (b) and Fig l.(d) ) that could make the 2 control surfaces of a Variable Geometry controller (on 2 different member 113 and 115 in Fig.1 ) to rotate (and stop) at different time during the turning of the servo shaft :
  • the upper part of (a) shows in the vertical dimension Rotl is the rotation of the 1 st surface of controls on member 113 of Fig 1(d). It (113) will turn first and when it reached close to 90 degree from initial position (which wide enough to allow 115 to expand).
  • Rot2 is the rotation of the 2nd surface of controls on member 115 of Fig 1(d). 115 begin to rotate "up" 90 degrees while 113 stops rotating shortly after 115 begins.
  • the timing and turning angle of the servo are exemplary as the servo turning angles and gear ratios can change. In the gear shown in Fig 3.
  • the "active/driving" gear (or the bevel gear) on the servo shaft are 1.5 times diameter to the "passive" gear, and each are partial tooth gear having at around 60 degree of tooth (other area are smooth or not exists) shown as Angl and Ang3, the phase difference between the 2 gears allow them to drive different parts(payloads) such as 113 and 115 at different timing and the gear ratio allow it to convert the servo turning angle (at here each phase around 60 degree) to the required turning angle of the payload (113 and 115) each around 90 degree (shown as Ang2).
  • the drive train may have a clutch-like on- demand engaging/disengaging mechanism to one or more of the partial (tooth) gear(s) so that the timing or movement of members can be adjusted (dynamically).
  • the "active/moving" members can be allowed to rotate in different phase/timing, and possible more than 90 degrees from original ("folded") position, (some of the forms might require slight changes to drive train such as but not limited to: adding a clutch-like on-demand engaging/disengaging mechanism to one or more of the partial (tooth) gear(s) ), which might be used in purpose such as (but not limited to) to form a shape simulating "mouse-like" operations, in which 113 can rotate to (for example but not limited to ) around 150 degrees so that the overall shape of the apparatus (and the support) feels like that of a mouse to index and other fingers, while 115 might be selectively folded or rotate to an "expanded" position (for example but not limited to ) around 120-150 degrees from the folding position so that thumb can reach the controls easily in the "mouse-like" configuration/form of the apparatus.
  • an apparatus for user interacting with computer system (such as but not limited to a game controller) comprised of:
  • Multiple "axis" of control such as joysticks and buttons/keys and/or touch areas on the surface of the control in a layout normally allows multiple controls being manipulated/operated simultaneously by different fingers, which designed for a certain (specific) holding pose/gesture (such as 2 hands on the handle with thumbs on the joysticks/keypads) which considered the "standard/normal” (by design) gesture (hand/finger pose) for this controller";
  • a touch sensitive area (such as but not limited to touch pad, touch screen) that is located on the surface of or articulated to said handheld/wearable device that can be conveniently reached by index finger or middle finger while user's hands/other fingers in corresponding "Normal/Standard" (game controller) operating gesture/ (pose of hand/fingers) ⁇ such as thumb “on the control”(such as the joy stick)-- and allowing the finger (index finger or middle finger) that perform the finger writing to be in a
  • one example (but not limited to) of said "normal'Vcomfortable finger writing position could be: the location of the touch area is within the range of the index and middle finger when user hold a "normal/standard operation pose" for normal
  • the possible locations (of the furthest point of the touching surface) are within the sphere which centered from the knuckle of a index finger(or the point index finger connects to the palm) (normal holding pose) and radius is the length of the index finger.
  • the "plane" of the touch pad/area being ergonometric arranged ⁇ for example at an angle to the plane of the thumb control (such as joystick), or in a modular design it could be add-on when needed to a convenient location / removed when not needed, or in a foldable/displace-able design it can "stick out/pop up'Vextracted to a convenient location allowing comfortable finger writing;
  • finger writing recognition modules (it could be for example but not limited to the same as an existing handwriting module) either on board the controller or on (host) computer system connected (such as but not limited to PC, tablet, smart TV or game console);
  • the said "finger writing” recognition modules process the touch/finger writing events/data from said touch sensitive area (and desirably also take into account other events/info generated by user at the same time such as (but not limited to) those defined as “function keys") and generate (or “translate to”)
  • key(s)/button(s)/touch sensitive area(s), which can be pressed/held/touched by other fingers when user performing the finger writing can be arranged on the (surface of) input device/controller, desirably arranged on a plane that is (usually) at an angle (such as but not limit to 90 degree) with the plane of the touch pad/area for finger writing, so that other finger(s) (including those used to hold the handheld device) can perform additional input (for example but not limit to :
  • additional keys/button for example but not limit to function keys such as Shift, Ctrl, Fn and etc
  • function keys such as Shift, Ctrl, Fn and etc
  • user's other finger can operate these keys/buttons while user performing the finger writing input (such as arranged on the side of the same controller where other fingers or thumb can easily reach while holding the controller so that user can operate with one hand
  • additional keys/button for example but not limit to function keys such as Shift, Ctrl, Fn and etc
  • it is desirable additional keys/button are on the same handheld device/wearable device and arranged in places user's other finger can operate these keys/buttons while user performing the finger writing input (such as arranged on the side of the controller(s) where fingers or thumb of the other hand can easily reach while holding the controller so that user can operate with both hands
  • finger writing input such as arranged on the side of the controller(s) where fingers or thumb of the other hand can easily reach while holding the controller so that user can operate with both hands
  • controller/input system to form the end output result (for example if user hold down key(s) on the side of the controller (such as defined as Ctrl or Fn key), the output result is like user hold down these keys and type on the keyboard)
  • a wearable controllers such as a glove like game controller could have controls and the touch area for finger writing input on 2 planes (one for index finger write input, one for thumb joystick like input) which are at angle to each other (such as but not limit to perpendicular), such as shown in Fig 5, (1) is the touch area for index finger for finger writing input and (2) is the touch area or joystick/button(s) for thumb to control direction, just like on a normal game pad/controller. (1) and (2) might be foldable/collapsible when they are not in use, so that they wont obstacle/hinder hand/fingers performing other activity/gestures.
  • a wearable device for tracking user gesture for example a glove (half glove)-like wearable with sensors (or beacon/makers or trackers) on it, also have a insert-able/modular or collapse-able touch input module (such as a touch pad/screen) that can be reached by user's index finger (optionally in an on demand manner);
  • a wearable device for tracking user gesture for example a glove (half glove)-like wearable with sensors (or beacon/makers or trackers) on it, also have a insert-able/modular or collapse-able touch input module (such as a touch pad/screen) that can be reached by user's index finger (optionally in an on demand manner);
  • the glove have other touch/press input areas or buttons so that other fingers can provide input simultaneously,.
  • the structure supporting the touch/press input interface could be folded/compressed so that when not needed, it will not interference other hand gesture recognition (such as grab); [00120] "finger writing" recognition modules either on board the glove-like wearable controller or on (host) computer system connected (such as but not limited to PC, tablet, smart TV or game console and etc.);
  • the said "finger writing” recognition modules process the touch/finger writing events/data from said touch sensitive area (and desirably also take into account other fingers movement/gesture such as those for “function keys") and generate (or “translate to") character(s)/symbol(s)/command(s) as user input (to the downstream system).
  • the finger writing can be interpreted in the context of the gesture by other fingers/palm or hand.
  • finger's gesture can be use to select candidates of the recognition.
  • a wearable device for tracking user gesture for example a glove (or half glove)-like wearable with sensors (or beacon/makers or trackers) on it for tracing user's hand movement and/or gesture, also touch input module (such as a touch pad/screen) located on the side of the thumb (same side as the hand back) that can be conveniently reached by user's index finger (or middle finger) of the same hand;
  • touch input module such as a touch pad/screen
  • finger writing” recognition modules either on board the glove-like wearable controller or on (host) computer system connected (such as but not limited to PC, tablet, smart TV or game console and etc.);
  • the recognition modules process the touch/finger writing events/data from the touch area (and desirably also take into account other fingers movement/gesture such as those for "function keys") and generate character(s)/symbol(s) as user input (to the downstream system)
  • a apparatus to for character/symbol input from a handheld or wearable interactive device/controller which allows multiple fingers to provide input (to for example but not limited to a computer system/host system such as but limited to game console/TV/setup box/audio system/appliances which the device/controller connected or controlling) conveniently (and possibly even with single-hand) that do not require the help from (user's) vision or voice comprised of: [00129]
  • a compact/limited size (such as but not limited to: stamp-sized, large enough for using finger writing methods to "draw'Vwrite one character, and desirably not too big -such as allowing more than 2 characters— to make it difficult to arrange/port on a controller for an ergonometric location) touch sensitive means/area (such as but not limited to touch pad, touch screen and etc.) that is on the surface of or articulated to said
  • the handheld/wearable device at an (ergonometric) location that can be easily reached by user's finger tip (or top part of the thumb) in a "normal'Vcomfortable hand/finger writing position; (the size of the touch sensitive area is no less than 6mm x 6mm, and maybe not need to be bigger than the size to accommodate a normal finger writing character/symbol);
  • touch area might have boundaries/(tactile) hints to indicate the effective touch sensitive area for finger writing input, such as but not limited to: protrusion/recess on the surface or raised/ depressed areas, edges, bumps and etc , it also could be surface (material) with different texture/tactile feeling than surrounding areas, so that user can distinguish the area using finger with tactile feeling alone without the need of visual (confirmation or help/aid to locate) ;
  • finger writing Means for sending "touch information/data" (“finger writing") collected from the sensor (such as but not limited to touch location/coordinates, events, patterns, force/magnitude (if the pad can provide) and etc) to finger writing recognition module which either onboard the handheld/mobile/wearable device/controller or on (host) computer system connected - if the recognition module is on board this controller the means could be as simple as wired connection, and if the recognition module is considered remote then means for remote connection (such as wired or wireless connection and related protocols) will be needed, and might need to add additional information packet to transmitting the touch events/data (such as by modifying/adding to existing communication protocol);
  • finger writing recognition module which either onboard the handheld/mobile/wearable device/controller or on (host) computer system connected - if the recognition module is on board this controller the means could be as simple as wired connection, and if the recognition module is considered remote then means for remote connection (such as wired or wireless connection and related protocols) will be needed, and might need to add additional information
  • Module for "finger writing” recognition (which for example but not limited to be basically same as handwriting recognition module) that process said "touch information/data" ("finger writing");
  • the said modules for "finger writing” recognition process the touch/finger writing events/data from said touch area (and desirably also take into account other events/info generated by user at the same time such as (but not limited to) those defined as “function keys") and generate (or “translate to") character(s)/symbol(s)/command(s) as user input (to the downstream system);
  • key(s)/button(s)/touch sensitive area(s), which can be pressed/held/touched by other fingers when user performing the finger writing can be arranged on the (surface of) input device/controller, desirably arranged on a plane that is (usually) at an angle (such as but not limit to 90 degree) with the plane of the touch pad/area for finger writing, so that other finger(s) (including those used to hold the handheld device) can perform additional input, such as but not limit to holding the defined "function keys” such as shift, Ctrl or Fn key, or perform selection for candidates (character, symbol) right after the finger writing, (and determine the output character/symbol or direction/command), for example if user pressing(holding) the shift of Fn key at the time performing the finger writing, the output will be the same as user type on the keyboard and press(holding) the shift or Fn key.
  • functions keys such as shift, Ctrl or Fn key
  • the recognition result might be combined with other input (if any) from user such as key/button status user holding/pressed or gesture user is making (at the same time user performing the finger writing) to form the result of the output character(s)/symbol(s) or command(s); for example if user pressing(holding) the shift of Fn key at the time performing the finger writing, the output will be the same as user type on the keyboard and press(holding) the shift or Fn key.
  • the data/events/information from said touch pad/area is transmitted to said finger writing recognition module in a "realtime or near-realtime” fashion such as but not limited to events/information "packets" about the location/coordinate of the current touch point and timestamp, or the "segment” of the current (continues) curve, or it could be the pattern when touch system determines that the whole character is inputted;
  • the apparatus might have/expose an interface (which output character/symbol) that is similar or simulate that of a keyboard interface or general HID interface that includes typing/character input, or a COM/stream interface;
  • a handheld or wearable pointing device/controller such as but not limited to a mouse or 3D mouse, 3D controller which provides
  • a compact/limited size (such as but not limited to: stamp-sized, large enough for using finger writing methods to "draw'Vwrite one character, and desirably not too big -such as allowing more than 2 characters— to make it difficult to arrange/port on a controller for an ergonometric location) touch sensitive means (such as but not limited to touch pad, touch screen and etc.) that is on the surface of or articulated to said
  • handheld/wearable pointing device at a "convenient reachable” that can be easily reached by user's (index or middle) finger tip in a "normal'Vcomfortable hand/finger writing position; (the size of the touch sensitive area is no less than 6mm x 6mm, and maybe not need to be bigger than the size to accommodate a normal finger writing character/symbol);
  • Such "convenient reachable" location of touch area /means is on (the surface) or articulated to said input device (such as but not limited to mouse, 3D mouse, 3D controller) that can be reached by index finger or middle finger and perform "finger writing" while maintaining a normal operation gesture the device; for example the touch area could be on the "left” or “middle” button of the mouse, or it could be in the area that is about the location of "middle button” or further up, so that left-handers can also operate such touch pad/area with ease;
  • said input device such as but not limited to mouse, 3D mouse, 3D controller
  • finger writing recognition module (which could be for example but not limit to the same or similar to handwriting recognition modules already available) on board the controller or "remote" on the (host) computer system connected in which case require transmitting the information/data/events of finger writing from the controller to the host computer system), it is desirable such recognition might take consideration of (if any) simultaneous input by other fingers - such as key/button press from additional keys on the controller such as but not limited to additional keys/function keys on side of the mouse/3D mouse or 3D controller which can be operated by user's other finger at the same time) or gestures detected by the controller/input system to form the end output result (for example if user hold down Ctrl or Fn key, the output result is like user hold down these keys and type on the keyboard)
  • additional keys on the controller such as but not limited to additional keys/function keys on side of the mouse/3D mouse or 3D controller which can be operated by user's other finger at the same time
  • gestures detected by the controller/input system to form the end output result (
  • the said "finger writing” recognition modules process the touch/finger writing events/data from said touch area (and desirably also take into account other events/info generated by user at the same time such as (but not limited to) those defined as “function keys") and generate (or “translate to") character(s)/symbol(s)/command(s) as user input (to the downstream system).
  • Game controller sending messages while playing, or easily input URL/text for browsing/searching
  • Smart TV User take a standard grabbing pose for the remote controller, and while watching TV, user can easily use index or thumb to input character on remote controllers equipped with this "finger writing" touch area/pad.
  • the touch/finger writing can be recognized by the recognition system, and provide text or command to downstream system/interface.
  • the touch system might also recognize swiping, zooming or other navigation command similar to that of the touch screen of a smart phone.
  • variable geometry hand-based controller/input apparatus comprised of movable parts that can form at least 2
  • one form/shape is the "fully expanded" mode/form that can accommodate user's hand and allow user to manipulate controls easily/ergonometrically (or : without using great efforts), and simultaneously.
  • the surface of a control stick /joy stick might be not on the same plane of some buttons/triggers, when the facing direction of each control is optimized for the finger, for example the "plane" of the surface of control for thumb it is at an angle (such as but not limited to perpendicular) to the palm and for index finger to operate simultaneously, the plane of surfaces for control operated by index finger (such as buttons /triggers) is at an angle to the 1 st plane of surface of control for the thumb (such as but not limited to around 90 degree), such as the situation in Xbox or PS.
  • the profile height of the apparatus (the maximum distance of all points on the apparatus to the plane of the palm) is no more than 60% of the "expanded'Vfunctional form.
  • user can (more easily) using their hands/fingers for other gesture (such as simulating a key press or swipe) or purpose (such as press a button/key on another device);
  • the shape/form change/transformation is driven by a driving mechanism (for example but not limited to servo(s) ), and it is desirable such change might be triggered or on-demand;
  • a driving mechanism for example but not limited to servo(s)
  • the driving mechanism can drive at least 2 members with planes/surfaces for hand based control (as mentioned above) to transform (such as but not limited to rotate/move/displacement, such as like rotating 90 degrees ) from collapsed/folding position/form to expanded/functional position, and might cause the 2 planes/surfaces to interlock to form a stable controller.
  • planes/surfaces for hand based control (as mentioned above) to transform (such as but not limited to rotate/move/displacement, such as like rotating 90 degrees ) from collapsed/folding position/form to expanded/functional position, and might cause the 2 planes/surfaces to interlock to form a stable controller.
  • variable geometry hand-based controller/input apparatus comprised of 2 part that moves on different axis(direction), so that 2 surfaces at an angle (such as but not limited to 90 deg) can be formed to accommodate user's thumb and index finger (and other fingers) respectively - user can reach controls on these 2 surfaces easily and comfortably without need to adjust held/grabbing
  • thumb and other fingers such as aiming with thumb and at the same time shooting/pull the trigger with index finger
  • the surface could be arched so that they feels more ergonometric and allow smaller "profile” when collapsed, just like Fig 1.
  • one (or more) limited-(distance/rotation)travel or activation mechanism such as a servo with drive train with partial (tooth) gears can be used to provide "2 (or more) phase" movement from one rotational action from an actuator (such as a servo 105)
  • phase 1 is open one surface for finger controls (such as 113) for the index finger, desirably at multiple (adjustable/customizable) angles
  • phase 2 is erecting/rotating up the "thumb control" surface, optionally phase 3 could be interlocking them together (powered or manually).
  • an ergonometric variable geometry hand- based controller/input apparatus comprised of: a 1st member with surface for hand/finger control , and 2 articulated member that can rotate along 2 different axis, one of the member might be ergonometric shaped to fit in to of the palm and desirable can be optionally attached to the hand (palm) and move together with the hand without relative movement, so that to said 1st member to rotate "against" the palm via this member thus allow the apparatus to transform to a different form/shape ("operational state") with different angles rotated, for purpose such as but not limited to: simulating the physical shape/surface/interface/grasp of different tools/handles of different sizes/shapes;
  • Another articulated member with surface for hand/finger control (“2nd control surface member”) could also rotate against said 1st member, and by itself can also have controls on it, so that to provide 2 surfaces with hand/finger control on them at an angle with each other (such
  • a latch/locking mechanism between the (moving) members which could be driven by an actuator or by hand to "lock” the relative position (such as when 2nd surface is fully expanded) between 2 or more members so that the control surface have a more "solid” feel when operated by the user.
  • position or motion tracking means such as optical mouse chip and lenses on the apparatus that is facing to the "desk” surface (or: the direction of what the palm is facing) when this "mouse mode" is used and provide movement

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Hardware Design (AREA)
  • Position Input By Displaying (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

A methods and apparatus of hand based "hybrid" input (human computer interface) allowing compact, efficient, even single-handed operable symbol and character input methods that do not rely on vision and utilizing skills user already have. Also about a new kind of dynamic/shape-transformable human-machine interaction means (methods and apparatus/controllers) ~ Variable Geometry hand-based controller, for providing on-demand, (Ergonometric and) change-able (transformable) interface for user's hand to interact with, thus provide more flexible/versatile way of interaction and possibly simulating (including the user experience) a variety of (hand-based) controllers/devices, so that user can have more than one way of interaction without need to switch to other controllers.

Description

Apparatus and Method for hybrid type of input of buttons/keys and "finger writing" and low profile / Variable Geometry hand-based controller
Cross Reference to Related Applications
[0001] The present application claims priority from US Provisional Patents
Application Serial Nos. 62/191528 and 62/210413 filed July 13, 2015 and August 26, 2015 respectively, the full disclosures of which are hereby incorporated by reference herein.
Technical Field
[0002] The present invention relates to hand based "hybrid" input device (human computer interface), more specifically related to methods and apparatus to allow compact, single hand operable symbol and character input methods that do not rely on vision and utilizing skills user already have.
[0003] A new kind of dynamic/shape-transformable human-machine interaction means (methods and apparatus/controllers) is also discussed here, more precisely relating to Variable Geometry hand-based controller, for providing on-demand, (Ergonometric and) change-able (transformable) interface for user's hand to interact with, thus provide more flexible/versatile way of interaction and possibly simulating (including the user experience) a variety of (hand-based) controllers/devices, so that user can have more than one way of interaction without need to switch to other controllers.
Background Art
[0004] With the development of computer and home entertainment system/smart appliances, there are a growing need for systems become connected/going online, so connect to website/services, input URL, search and filling forms for these system (such as a game console) are common place, however current game controller or many other smart appliances/smart TV does not provide an easy to use/convenient character/symbol input mechanism.
[0005] It is known in the prior art hand based controllers such as joystick, mouse, game pads, touch pads, keyboard, data gloves, wii motes and etc. are commonly used to (and come in pair) with one or more machine being controlled. Such controller usually have controls on surface of the controller such as buttons, touch pads, joysticks and haptic feedbacks , and the layout usually customized to the specific interaction between user and machines, for example, the frequent used controls are made bigger and easy to reach by the fingers.
[0006] However, with the introduction of VR and requirements for interaction conveniently with multiple devices, such "specialized" controller are not flexible enough to allow controlling different devices effectively/more convenient.
[0007] A suggested solution provided in this invention, by using a unique variable (change-able) geometry hand operated apparatus that can change/transform between multiple geometry shapes/forms, with at least one of these form is functional for hand/finger manipulation (of the controls on the device/controller), thus to provide variable interfaces to user's hand and allow "on-demand"/dynamic changing of the interfaces, thus allow much more flexible manipulation and integration of multiple controls/devices easily without need to physically change the controller in user's hands, and provide user with unique experiences while still take advantage of the manipulation skills for the controls they already have.
[0008] A low-profile joystick like control from using touch sensitive area (pad, screen) is also discussed here. (Optionally) It can be used in the Variable Geometry hand- based controller (as controls for user to interact with).
Summary of the Invention
[0009] Definitions. As used in this description and the accompanying claims, the following terms shall have the meanings indicated, unless the context otherwise requires:
[0010] Finger writing: A pattern (or a sequence of touch events/strokes) on a surface touched/draw by the tip of a finger (such as index finger, middle finger or top part of the thumb), ~ basically characters, symbols, or navigation (such as swipe, rotate) "marks" written on the surface by using finger tip (or top part of the thumb). The resulting pattern could be similar to that of a handwriting, however finger writing does not need to have a pen or stylus involved and only need one finger to draw the pattern, (could also mean the act of drawing such pattern)
[0011] surface facing direction for "normal'Vcomfortable hand/finger writing: [0012] In a usual/normal holding pose(s) of a controller, in order to comfortably perform "finger writing" , usually the surface on which the finger (or thumb) is writing on is required to be at least partially towards the user ~ such as (but not limit to) facing up or facing sideway but in a direction not "facing away" from user . (For example in the 180 degree range that is considered partially facing the user.) Such direction allow relatively comfortable position for finger (or the thumb) to "write" on the surface with some accuracy. (The corresponding hand position for this "normal/comfortable" hand/finger writing position, when using index finger or middle finger is palm facing downwards, or facing sideway that is not considered towards user himself/herself, while for thumb writing in some situations the hand position is for the palm to be partially upwards or partially towards user, and not quiet desirable for facing away from user).
[0013] Normal/Standard (game controller) operating gesture/ (pose of hand/fingers):
[0014] Game controller for user interacting with computer system/game console usually have multiple "axis" of control such as with joysticks and buttons/keys and/or touch areas on it,
[0015] the "Layout" of the controls (buttons, joysticks, touch pads) normally allows multiple controls being manipulated/operated simultaneously by different fingers, and user have a "normal" holding pose/gesture (such as 2 hands on the handle with thumbs on the joysticks/keypads) are considered the "standard" or "normal" (by design) posture/gesture for playing the game". The new touch pad finger writing input method in this invention is also based on such standard/normal gesture(pose of hand/fingers), and allow index/middle finger to do text input without (significantly/substantially) interrupting other finger's operation.
[0016] Hand/finger-operated-controls: Controls with size suitable to be
operate/manipulated by user's finger/thumb, such as but not limit to button(s), key(s), joystick(s) (especially gamepad like), touch sensitive areas such as but not limited to (touch pad/touch screen and etc).
[0017] Surface for hand/finger control:
[0018] A surface of a controller with one or more Hand/finger-operated-controls on it for user interaction. [0019] Partial (tooth) gears - gears (including bevel gears) or pair of gears with only a part of the whole arc having tooth, other part are either smooth (and not driving the movement of the counterpart/meshing gear) or does not exists.
[0020]
[0021] In a first embodiment of a method to allow convenient (and even with single- hand) character/symbol input from a handheld or wearable interactive device/controller that allows multiple fingers to provide input (to for example but not limited to a computer system/host system such as but limited to game console/TV/setup box/audio
system/appliances which the device/controller connected or controlling) that do not require the help from (user's) vision or voice includes:
[0022] Providing user with a compact/limited size(such as but not limited to: stamp- sized or an area large enough for accommodating using finger writing methods to
"draw'Vwrite one character/symbol, and desirably not too big -such as bigger than an area allowing more than 2 of such characters/symbols to be drawn ~ touch sensitive means (such as but not limited to touch pad, touch screen and etc.) that is on the surface of or articulated to said handheld/wearable device at an (ergonometric) location that have a surface facing direction for "normal'Vcomfortable hand/finger writing, and can be easily reached by user's finger tip in a "normal'Vcomfortable hand/finger writing position; (the size of the touch sensitive area is no less than 6mm x 6mm, and maybe not need to be bigger than the size to accommodate a normal finger writing character/symbol); (It is desirable said touch area might have boundaries/(tactile) hints to indicate the effective touch sensitive area for finger writing input, such as but not limited to: protrusion/recess on the surface or raised/ depressed areas, edges, bumps and etc , it also could be surface (material) with different texture/tactile feeling than surrounding areas, so that user can distinguish the area using finger with tactile feeling alone without the need of visual (confirmation or help/aid to locate) );
[0023] Sensing user's "touch information" ("finger writing") on the touch sensitive area such as but not limited to touch location/coordinates, events, patterns, force/magnitude (if the pad can provide) and etc.,
[0024] Sending info/data collected from the sensor to finger writing recognition module, either onboard the handheld/mobile/wearable device/controller or on (host) computer system connected; [0025] Recognizing the "finger writing" from said data/info collected;
[0026] And it is desirable the recognition result might be combined with other input (if any) from user such as key/button user holding/pressed or gesture user is making (at the same time user performing the finger writing) to form the result of the output
character(s)/symbol(s) or command(s); for example if user pressing(holding) the additional keys on said controller (such as those could be defined as function keys such as "shift" or "Fn" key) at the time performing the finger writing, the output will be the same as user type on the keyboard and press(holding) the corresponding function key (such as shift or Fn key) on the keyboard;
[0027] Outputting the result of the recognition (such as character(s)/symbols or commands) to a down stream system/interface (such as but not limited to a text or keyboard receiving interface, or GUI elements such as a text/URL/search input box, F E(input method editor) window, and etc. ).
[0028] In a related embodiment, it is possible the data/events/information from said touch pad/area is transmitted to said finger writing recognition module in a "real-time or near-real-time" fashion such as but not limited to events/information "packets" about the location/coordinate of the current touch point and timestamp, or the "segment" of the current (continues) curve, or it could be the pattern when touch system determines that the whole character is inputted;
[0029] In a related embodiment, (optionally) user can be prompt (visually or audibly) for candidates of recognition, or auto-completion;
[0030] In an related embodiment, the method further includes combining information from buttons/keys/touch sensitive areas operable by finger(s) other than the one used for handwriting, so that user can use the combination of keys (together with the handwriting) for more complicated and convenient input, such keys/buttons can provide functions such as CAPs, Shift, control, Fn, symbols, space, return and other not "visible" characters/functions, or allow user to quickly select from candidates/auto completion.
[0031] In a related embodiment, it is desirable to combine other key/button press events or gestures (such as but not limit to those of other fingers) that user performed simultaneously (from the same hand or the other hand), for determining the output character/symbol or direction/command; for example if user pressing(holding) the shift of Fn key (by the fingers on the same hand or the other hand, if applicable such as in case both hands have controllers) at the time performing the finger writing, the output will be the same as user type on the keyboard and press(holding) the shift or Fn key.
[0032] In a related embodiment, the output character/symbol might be in a interface similar or simulating to that of a keyboard interface;
[0033] In another embodiment (of a method) to allow convenient (and even with single hand) character input from a (or "a pair of in the controller is a pair for both hands) handheld or wearable interactive device/controller that allows multiple fingers to provide input (to the computer system it connects to) simultaneously such as but not limit to a game controller without interfering existing functionalities/finger operations/game play (such as the ability to operate the joystick with the thumb, and other buttons) includes:
[0034] Providing user a compact/limited size(such as but not limited to: stamp-sized, large enough for using finger writing methods to "draw'Vwrite one character, and desirably not too big -such as allowing more than 2 characters— to make it difficult to arrange/port on a controller for an ergonometric location) touch sensitive means (such as but not limited to touch pad, touch screen and etc.) that is on the surface of or articulated to said
handheld/wearable device and the touch area have a surface facing direction for
"normal'Vcomfortable hand/finger writing and at a "convenient reachable" location that can be easily reached by user's finger tip in a "normal'Vcomfortable hand/finger writing position;
[0035] Such "convenient reachable" location of touch area /means is on (the surface) or articulated to said input device (such as but not limited to the game controller) that can be reached by index finger or middle finger and perform "finger writing" while maintaining a normal operation gesture for other controls/buttons on the input device/controller (such as a game controller) - in another words, the location of the touch area is within the range of the index and middle finger when user hold a "normal operation status" for normal
input/manipulation when not doing text (finger writing) input— the possible locations (of the furthest point of the touching surface) are within the sphere which centered from the knuckle of a index finger(or the point index finger connects to the palm) (normal holding pose) and radius is the length of the index finger,
[0036] Performing "finger writing" recognition (by recognition modules on board the controller or transmitted the information/data/events of finger writing from the controller to the (host) computer system connected, such as but not limited to PC, tablet, smart TV or game console and perform recognition there), it is desirable such recognition might take consideration of (if any) simultaneous input by other fingers or gestures detected by the controller/input system to form the end output result (for example if user hold down Ctrl or Fn key, the output result is like user hold down these keys and type on the keyboard);
[0037] Outputting the result of the recognition (such as character(s)/symbols or commands) to a down stream system/interface (such as but not limited to a text or keyboard receiving interface, or GUI elements such as a text/URL/search input box, F E(input method editor) window, and etc. ).
[0038] In a related (optionally) user can be prompt (visually or audibly) for candidates of recognition, or auto-completion;
[0039] In a related embodiment, said touch area pad might be modular and can be added/removed from the controller.
[0040] In a related embodiment, said touch area pad might be
"foldable'V" collapsible" or can be displaced so that when user do not need to finger writing input, the touch area/pad can be "folded/collapsed" or removed from the "operating position" to prevent it from obstacle/hindering normal operation of fingers to the controls (such as game pad buttons/joysticks) .
[0041] In another embodiment (of a method) to allow convenient (and even with single hand) character input from a handheld or wearable pointing device/controller such as but not limited to a mouse or 3D mouse, 3D controller which provides positional/movement input (to the computer system it connects to) without interfering existing functionalities/ operations/navigation includes:
[0042] Providing user a compact/limited size(such as but not limited to: stamp-sized, large enough for using finger writing methods to "draw'Vwrite one character, and desirably not too big -such as allowing more than 2 characters— to make it difficult to arrange/port on a controller for an ergonometric location) touch sensitive means (such as but not limited to touch pad, touch screen and etc.) that is on the surface of or articulated to said
handheld/wearable pointing device and the touch area have a surface facing direction for "normal'Vcomfortable hand/finger writing and at a "convenient reachable" location that can be easily reached by user's finger tip in a "normal'Vcomfortable hand/finger writing position; [0043] Such "convenient reachable" location of touch area /means is on (the surface) or articulated to said input device (such as but not limited to mouse, 3D mouse, 3D controller) that can be reached by index finger or middle finger and perform "finger writing" while maintaining a normal operation gesture the device; for example the touch area could be on the "left" or "middle" button of the mouse, or it could be in the area that is about the location of "middle button" or further up, so that left-handers can also operate such touch pad/area with ease;
[0044] Performing "finger writing" recognition (by recognition modules on board the controller or transmitted the information/data/events of finger writing from the controller to the (host) computer system connected), it is desirable such recognition might take consideration of (if any) simultaneous input by other fingers - such as key/button press from additional keys on the controller such as but not limited to additional keys/function keys on side of the mouse/3D mouse or 3D controller which can be operated by user's other finger at the same time) or gestures detected by the controller/input system to form the end output result (for example if user hold down Ctrl or Fn key, the output result is like user hold down these keys and type on the keyboard);
[0045] Outputting the result of the recognition (such as character(s)/symbols or commands) to a down stream system/interface (such as but not limited to a text or keyboard receiving interface, or GUI elements such as a text/URL/search input box, F E(input method editor) window, and etc. ).
[0046]
[0047] In a first embodiment of a Variable Geometry hand-based control apparatus comprised of:
[0048] At least 2 articulated or touched members that can rotate/move/displacement against each other and at least one of the members can fit into the operation range of a user's hand/fingers (such as but not limited to: the size fits into user's palm), and one or more controlled actuator/powered driving mechanism on the apparatus and operationally connected/coupled to at least one(or 2) members that is able to rotate/move/displacement at least one "active/movable" member rendering the geometry form/shape change from to an other following control from a controller for form/shape control (which desirably could , either coordinates with or controlled by or can processing request from, appropriate user interface(s) so that the current form of the apparatus is consistent with the current (intended) user interface for proper interaction requirements/preferences which could be different for different apps/software that needs interaction/input from user);
[0049] there is a least one "surface for hand/finger control" (with hand/finger operated controls on it) located on least one of said "active/movable" member(s) which is(are) operationally connected to the driving mechanism and can move/rotate/displace (relative to other member(s));
[0050] In at least one of the geometry form/shapes of said hand-based control apparatus allows user to use hand/finger to operate the controls on the "surface for hand/finger control"— such as but not limited to: performing control of machine connected to the apparatus, performing input/interaction such as but not limit to touch input, pressing keys/buttons, manipulating joystick and etc.;
[0051] In at least one another geometry form/shapes of said hand-based control apparatus allowing user to performing hand-based operation(s) that is in a different manner with the previous one, for example but not limited to using gesture input (without any buttons or touch inputs) or at least using a group of controls that is not entirely the same as the previous one (which means the hand-operable controls in this form could be a subset or supper set of that in the previous ones or it could include different controls/input means);
[0052] So that by switching/changing to different geometry forms/shapes, the apparatus allows user to perform control/interaction with a computer implemented system (that expects/requires user interaction) in different manner/form without the need to change to another controller, and might also be able to simulate the physical experience of different kind of controls.
[0053] In a related embodiment the apparatus either
[0054] 1) comprised of least 2 members having surface for hand/finger control and at least one of the members can be moved/rotated/displaced by said actuator/driving mechanism (against others),
[0055] or 2) comprised of least 3 (movable) members that can move/rotate against each other and at least one surface for hand/finger control exists on a movable part driven by said actuator/driving mechanism. [0056] In a further related embodiment, it is desirable such change of form/shape can be performed dynamically or "on-demand" according to the requirement of the current interaction (for example but not limit to: when changing from one controller to an other controller, when simulating different controllers for different devices, and etc.)
[0057]
[0058] In a related embodiment where the apparatus comprised of at least 3
(movable) members that can move/rotate against each other, the members might be moved in a different time sequence and not always move at the same time.
[0059] In a further related embodiment, said movement of members has a sequence of movement with least 2 timing/phase(or step) for movement of members or transformation (members move at different timing/phase of a sequence of movement, similar to what is depicted in Fig.3 a );
[0060] In a further related embodiment, at least 2 different timing/step/phase of operation/movement in said sequence of are driven by a common actuator (such as but not limited to servo) using a drive train (such as but not limited to shaft and transmission system such as partial (tooth) gears) that is capable of turning actuator output at different timing/phase (such as but limited to: rotational angle, travel distance and etc) into movement of different members (at different time) (for example but not limited to the drive train system depicted in Fig l.d and Fig 3).
[0061] In a related embodiment it is desirable at least 2 surface for hand/finger control can be formed at an angle to each other and both within hand/finger reach at the same time so that user can use different finger/thumb to operate different controls on these surfaces at the same time.
[0062] In a related embodiment, it is desirable (at least) one of these form is a low- profile/non-blocking/minimum-interference form (such as but not limited to the form depicted in Fig.1 (c ) ) to allow user to use hand/fingers for other purpose/operations (such as but not limit to gesture input, pressing other buttons or operate other devices without the need for removing/releasing this apparatus from hand), and at least one form/shape is a "functional" form where the controls on the "surface for hand/finger control" can be reached conveniently and manipulated. [0063] In a related embodiment, The mechanism can rendering (at least) 2 planes (one for thumb/joystick) and one for index/other fingers to change (such as rotate, like 90 deg ) from collapsed position to expanded position, and might cause the 2 planes interlock to form a stable controller.
[0064] In a related embodiment, the at least one of the folding axis of a moveable member is (substantially) parallel to that of the average folding axis (vector) of the fingers of a user's hand excepting the thumb.
[0065] In a related embodiment, there could be a controller have finger/hand- operable controls on the surface for the other hand of user that can use together with this Variable Geometry hand-based control apparatus to form a full controller;
[0066] In a further related embodiment is desirable the 2 controller can be attached/coupled together when the Variable Geometry hand-based control apparatus is in (some) stable form(s) such as but not limited to when it is expanded to a "full grip" mode, so that for example user can use the attached/coupled controllers with both hands as "one piece", maybe (but not limited to) like a standard gamepad.
[0067] In a further related embodiment the for the "secondary" controller in the hand other than the one using the Variable Geometry hand-based control apparatus, the finger- operable controls on the surface of the controller and the "layout" is similar to the
"secondary side" (usually "left side") of a game pad, maybe depends on user is right-handed or left-handed (in which case it is the right side of a game pad, and Variable Geometry hand- based control apparatus to form a full controller is in user's left hand);
[0068] In one embodiment, it is desirable a control stick or joy stick having at least one driving mechanism (such as but not limited to an actuator connected to a shape /form control system) in addition to normal control stick/joystick components, and at least one articulated/touched "active/movable" member(s) having hand/finger-operable control(s) on it (such as but not limited to touch sensitive area such as touch pad/screen, buttons, knobs and etc.) that can be driven by said driving mechanism to move/rotate (such as but not limited to "rotate up" or "pop up") to a position/location that is easily reachable by user's fingers/hand, desirably on demand or dynamically following the control of the shape/form control system, so that user can operate the controls on said "active/movable" members simultaneously when manipulating the control/joy stick and/or the "original" controls on the stick. [0069] In a related embodiment, the controls on the "active/movable" members including a touch pad or touch screen which operationally connect to a touch and finger/hand writing recognitions system so that user can use the touch pad/screen to perform text input via finger/hand writing recognition system while at the same time operate the control/joy stick as well as other controls on it.
[0070] In one embodiment, it is desirable a game controller (such as a gamepad or Wii mote like controller) or remote controller having at least one driving mechanism (such as but not limited to an actuator connected to a shape /form control system) in addition to normal control stick/joystick components, and at least one articulated/touched
"active/movable" member(s) having hand/finger-operable control(s) on it (such as but not limited to touch sensitive area such as touch pad/screen, buttons, knobs and etc.) that can be driven by said driving mechanism to move/rotate (such as but not limited to "rotate up" or "pop up") to a position/location that is easily reachable by user's fingers/hand, desirably on demand or dynamically following the control of the shape/control system, so that user can operate the controls on said "active/movable" members simultaneously when manipulating the game controller/remote and the "original" controls" on said game controller/remote.
[0071] In a related embodiment, the controls on the "active/movable" members including a touch pad or touch screen which operationally connect to a touch and finger/hand writing recognitions system so that user can use the touch pad/screen to perform text input via finger/hand writing recognition system while at the same time operate the game controller/remote as well as other controls on it.
[0072] In a related embodiment of variable geometry hand-based control it is desirable the apparatus is wearable (or: can accommodate user's hand and/or arm and move together) such as but not limit to in a shape of glove/half glove or have strap on it (such as on member 112 in Fig 1) so that it can be secured to the user's hand/palm/wrist/arm (the limb/part of body considered wearing devices and can perform control), or have means for attaching to/coupling with a glove/wearable device and move together with hand or arm.
[0073] In one embodiment of variable geometry hand-based control that is capable of simulating the user experience (including tactile) of multiple of controllers/hand held device/tools, the apparatus can change/transform between multiple (stable) geometry forms simulating the size of the tool and desirably also (at least in some of the forms) the control location relative to hand (similar to that of the original tool/device being simulated), for example (but not limited to : ) , with the movement of at least one driving mechanism (such as but not limited to the rotation or travel of a servo) that drive(s) at least one member to move/rotate/displace so that the geometry from of the device changing from low profile "storage'Vminimum blocking form, to a form simulating small handler (thin handle) such as a dagger (with slightly bigger profile, this can be done in a way for example but not limited to: in Fig.1 control the servo 105 to rotate a small degree such as 15 or 20 degree comparing to the fully expanded degree in this case 60 degree), fishing rod, golf club; or the geometry of the apparatus can be controlled to change to a form to a medium "grip" (with even bigger profile, this can be done in a similar way for example but not limited to: in Fig.1 control the servo 105 to rotate a "moderate" or "medium" degree such as 30 or 40 degree compare to the fully expanded degree of 60 degree) , to a full grip (fully expanded, such as like those grip of for game pad); In a related embodiment it is also possible that the member can be driven to one or more "overreaching" form (bigger than "full" grip) which can be done for example but not limited to: slightly change the "drive train" in Fig.1 so that gear(s) or partial (tooth) gears (such as 125) responsible for driving another member 115 can be selectively disengaged ( and re-engaged) by (additional) means such as but not limited to a clutch-like means/mechanism that is controlled by the form/shape control system or by user, so that member 115 can be selectively rotated (not stay still) according to the "form" requirement; With 125 disengaged and 115 not rotated, the control system control the servo 105 to rotate a bigger degree (such as 90 or 120-150 degree) comparing to the fully expanded degree in this case 60 degree, so that the member (in this case 113) can form an angle 120-170 degrees from the palm (or member 112 that stay with the palm), this will form a mouse like shape/form (when user palm face down) that supports user's palm and at the same time allowing user to click controls such as 116 on the member 113 in a similar way of clicking a mouse button; optionally it is desirable the member 115 can also be rotated to a bigger angle than that of "full grip" of 90 degree, for example but not limit to around 120 - 150 degree, so that user can also use thumb to operate some controls on member 115 which enhance the function of the "mouse" form;
[0074] In a further related embodiment, further includes at least one optical position tracker or motion detector or similar motion/position tracking means, such as but not limited to a camera or camera-like optical pattern detection device such as that of a optical/laser mouse so that user's hand movement against a surface can be tracked/detected without the need of external sensors; said motion/position tracking means desirably is located on the "palm" side of the hand such but not limit to on or close to member 112 or 113 (in overreached form);
[0075] B) (Relate to) low-profile adjustable joystick (apparatus and related methods) [0076] Another method for variable geometry controller to simulating a physical user experience of a joystick's including: using touch sensitive area (such as but not limited to a touch pad or touch screen) together with a elastic film on top of the (primary) touch area so that user can feel the force from their fingers when the control is not centered and the direction and aptitude of the force indicate (by tactile sense) the "drift" or "distance" from the center.
[0077] In one related embodiment of a simulated joystick apparatus comprised of: a touch area (pad, screen) input, a thin elastic film which can be added on top of the touch area (such as but not limited to by means of attaching/coupling) , and it is desirable optionally the elastic film have protrusion/button/knob at or close to the center of the control/touch area, or tactile markings/patterns such as on the surface facing user or some form of tactile pattern so that user can feel the difference when finger touching different areas (which help him/her to use tactile sense only to locate or find center of the control without having to look at it; So that when user press the elastic film (by finger or by pressing central button/knob if available) a small part of the corresponding film in turn touches the touch sensitive area, and the touch sensitive areas (such as but not limit to touch pad, touch screen) can detect the "touch point(s)" x, y coordinates which can be translated into joystick x,y input by comparing the x, y coordinates of the touch point with the projection of joystick center position (of the joystick) in touch pad coordinates - this can be done for example but not limited to: by subtracting the "touch point" x, y coordinates with the "joystick center projection point" x, y coordinates, and maybe performing additional
scaling/direction/orientation correction and etc. , so that a joystick can be simulated with a low profile way not much higher than a touchpad.
[0078] In a related embodiment it is also desirable Teflon or other with resistance reduction coating can be applied to the other side of the film which facing the touch area surface so that to make it easy to move(drag) around the touch area (such as touch pad/screen) with user's finger pressed the elastic film or pressed the central button/knob.
[0079] So that when user operate this input the resistance force will reflect how far it have traveled/deviate from the center, and when user release it, it can automatically centered.
[0080] In an embodiment of related apparatus (as shown in Fig.2) the elastic film (such as but not limited to rubber) can be placed in a frame, such film can have a
docking/coupling mechanism (such as but not limited to mechanical coupling means such as screw, lip and etc. or magnetic coupling mechanism such as magnets or other methods such as sticky surfaces (heat sensitive glue and etc) on the frame of the film and corresponding counterparts on/around the touch sensitive area) that fix/securing the film on top of (and cover a least a part of) the touch sensitive area(such as pad/touch screen) (of a controller) to form a (low profile) joystick, while it can also be removed when necessary so that user can directly operate the touch sensitive area;
[0081] In a related embodiment, a knob/protrusion(s)/button (which maybe in a shape and size similar to that of a gamepad joystick's header part that touch's user's ginger/thumb) can be placed on the "center (area)" of said elastic film so it feels like a joystick to the finger and can be operated in a similar way.
[0082] In a related embodiment, there could be exchangeable films (or together with the frame) with different elasticity of the film, so that user can customize/adjusted the control to a comfortable/desirable level; In a related embodiment, the touch sensate area( or size) of the pad can also be (software) adjusted (such as but not limited to adjusting the "scale" when converting from touch pad x,y coordinates to joy stick x, y input as mentioned above) so that to accommodate different needs of user, generally speaking a larger area allows more accurate movement/aiming while a smaller area allows more nimble/fast movement/turning input.
[0083] In a related embodiment there could be different methods to secure the film and it frame to the touch sensitive area , for example but not limited to place on flat (like Fig 1. a and it desirably able to change between different "style" of film such as having different tactile pattern on it) or using "folding'Vturning like structure, as depicted in Fig 2 (b)
[0084] It may worth noting that although in some figures showing the film and the touch area having a similar shape and size, in implementation they do not have to be the same (or very similar) shape, for example but not limited to: the shape of the film can be round and the shape of the touch sensitive area (such as pad or touch screen) could be square/rectangular (maybe with round edges), and generally the size of the touch sensitive area is about the same or larger than that of the film. It is possible that the film only cover a part of the touch sensitive area (especially in the situation of a touch screen where only a part of the screen is "allocated" for the purpose of "joystick like input simulation". There could be different kind of couplings to secure the film (other than shown in the fig.2, for example a round to square/rectangular "adapting" frame can be used to secure a round shaped film to a rectangular touch sensitive area (using coupling means such as but not limit to mechanical, attach/adhesive (heat sensitive adhesive), magnetic (using magnets) and etc.)
[0085] This a low profile joystick control can be used in conjunction or as a control in the Variable Geometry hand-based control apparatus as described above.
[0086] In a related embodiment to The Variable Geometry hand-based control (such as but not limited to as those depicted in Fig. 1) , the control used on at least one of the Surface for hand/finger control (such as but not limited to 110 in Fig.1) is a low profile joystick control as described above.
Brief Description of the Drawings
[0087] The foregoing features of the invention will be more readily understood by reference to the following detailed description, taken with reference to the accompanying drawings, in which:
[0088] Fig 1 shows one kind of Variable Geometry Controller that can be combined with a hand wearable device (not shown). A "surface for hand/finger control" 113 with controls such as 116 in (b) that can be operated by fingers (such as but not limit to index finger) of user's hand, and it have multiple (showing 2) articulated members 112 and 115 which can (relative) rotate along axis 101 and 103 respectively. 112 can attach to a wearable device/harness and placed in user's palms and it could be in a form that curved along the general/average direction of "palmer crease(s)" or "folding seam" of a user's hand (102) that allow the upper part and lower part of user's hand to fold/ relative rotate.
[0089] 115 is a second "surface for hand/finger control" with controls such as touch pad/joystick 110 and buttons 111 on it and can be rotated between a operation/functional position/configuration as shown in (b) and a low-profile/ non-blocking position in (a) or (c). 113 can also rotate between an operational form/configuration such as (a) or (b) to a low- profile/non-blocking position/shape/form like in (c). The changes can be driven by actuator / driving mechanisms such as 105 (a servo) and related drive trains (such as but not limited to shaft and transmission system such as partial (tooth) gears) depicted in the figure.
[0090] Fig.1 (c) shows the "folding/collapsed" form/shape of the control which form a low profile, non-blocking shape that allow user to perform other operations/input with hand/fingers such as using gesture or press buttons/touch pad of other device.
[0091] Fig 1 (d) shows an example of driving mechanism where actuator (such as a servo) 105 is driving a (transmission) shaft 121 trough bearings 122 that is fixed on member 112, the shaft rotates around axis 101 and drive partial (tooth) gears (pair/train) 123 that cause member 113 which fixed to 105 to rotate against 112 (which usually have no relative movements against user's palm). Similarly, (at a different time/phase of rotation of 105 while 113 is "opened" at a sufficient angle (such as but not limited to more than 75 degrees in order to allow the other member 115 to be rotated upwards from 112), the partial (tooth) bevel gear pair 125 drives shaft 126 through bearings 127 fixed on 113 and cause it to rotate along axis 103 and through supporting structures such as but not limited to 128 which fixed on member 115 to cause it to rotate (raise up and down) and change between a folded form and a expanded form where user's thumb can operate it comfortably like Fig 1. (b). It is worth noting that the partial (tooth) gear (some part have tooth and some part are smooth and not "driving" the counter part gear, such as but not limited to those depicted in Fig 3 (b) ) and the configuration of it allows 112 and 115 to be turned at different time/phase of the turning of 105 and usually not actuated at the same time (while there could be overlapping).
[0092] Fig.2 shows a low profile joystick like control can be formed by adding a film of elastic on top of a touch area 201, and a frame of the film (and related coupling means in 205 and 202) can be used to secure the film on top of the touch area and desirable removable/exchangeable.
[0093] Fig 2 (a) shows different "style" of elastic films 203 and 203' (such as those with different tactile patterns as shown in 203', or some film have a central button/knob 204 like the top/head part of a joystick) can be placed in a frame 205 or 205' . Fig 2(b) shows a rotation/folding way of securing the film. [0094] Fig. 3 shows controller (such as remote control for smart TV/smart appliances) 301, it has a touch surface/area for finger writing input such as on top left area (shown as 303) operable for both index finger and thumb, that can be reached by index or middle finger or thumb for finger writing input for character/symbol.
[0095] Also there could be (optional) side buttons 302 which can be defined (such as function keys such as but not limited to Ctrl, shift,alt, Fn) and use by other finger(s).
[0096] Fig 4 shows A mouse (or 2D or 3D position/motion device) 401, can have a touch surface/area for finger writing input such as on left button (shown as 403 A), or in the middle section shown as 403B "above"(or to the upper side of) middle button/roller 402, that can be reached by index or middle finger for finger writing input for haracter/symbol.
[0097] Also there could be (optional) side buttons 405 which can be defined (such as function keys such as but not limited to Ctrl, shift,alt, Fn).
[0098] Fig 5 shows a wearable controller (such as a glove like game controller) could have (foldable/modular) controls and the touch area for "finger writing input" on 2 planes (one for index finger write input, one for thumb joystick like input) that at angle to each other (such as but not limit to perpendicular), (1) is the touch area for index finger for finger writing input and (2) is the touch area or joystick/button(s) for thumb to control direction, just like on a normal game pad/controller. (1) and (2) might be foldable/collapsible when they are not in use, so that they wont obstacle/hinder hand/fingers performing other activity/gestures.
[0099] Fig. 6 shows A typical game controller 601, which could have one or more joystick 602 which can be operated by thumb, and could have a touch pad area 605 for directional input. It can also have a touch surface/area for finger writing input such as on top left area (shown as 603B), or it could be pop up, foldable or "add-on" (modular) type as shown in 603A , both positions can be reached by index or middle finger for finger writing input for character/symbol.
[00100] Also there could be (optional) side buttons which can be defined (such as function keys such as but not limited to Ctrl, shift,alt, Fn).
[00101] Fig 7 shows a single hand type game controller 701 which include position tracking/motion sensing, it can also have a touch surface/area (such as a tich pad) 702 for finger/thumb writing input, or similar to Fig.6 it can also have be pop up, foldable or "add-on"(modular) type touch pads , and optionally side buttons which can be defined (such as function keys such as but not limited to Ctrl, shift,alt, Fn).
[00102] Fig. 8 shows different timing/phase (such as in Fig.8 (a) ) of the partial (tooth) gear (as shown in Fig 8 (b) and Fig l.(d) ) that could make the 2 control surfaces of a Variable Geometry controller (on 2 different member 113 and 115 in Fig.1 ) to rotate (and stop) at different time during the turning of the servo shaft : The upper part of (a) shows in the vertical dimension Rotl is the rotation of the 1 st surface of controls on member 113 of Fig 1(d). It (113) will turn first and when it reached close to 90 degree from initial position (which wide enough to allow 115 to expand). In the lower part of (a) shows in the vertical dimension Rot2 is the rotation of the 2nd surface of controls on member 115 of Fig 1(d). 115 begin to rotate "up" 90 degrees while 113 stops rotating shortly after 115 begins. The timing and turning angle of the servo are exemplary as the servo turning angles and gear ratios can change. In the gear shown in Fig 3. (b) the "active/driving" gear (or the bevel gear) on the servo shaft are 1.5 times diameter to the "passive" gear, and each are partial tooth gear having at around 60 degree of tooth (other area are smooth or not exists) shown as Angl and Ang3, the phase difference between the 2 gears allow them to drive different parts(payloads) such as 113 and 115 at different timing and the gear ratio allow it to convert the servo turning angle (at here each phase around 60 degree) to the required turning angle of the payload (113 and 115) each around 90 degree (shown as Ang2).
[00103] In a related embodiment the drive train may have a clutch-like on- demand engaging/disengaging mechanism to one or more of the partial (tooth) gear(s) so that the timing or movement of members can be adjusted (dynamically).
[00104] It is also possible (that in some embodiments), the "active/moving" members (such as 113, 115) can be allowed to rotate in different phase/timing, and possible more than 90 degrees from original ("folded") position, (some of the forms might require slight changes to drive train such as but not limited to: adding a clutch-like on-demand engaging/disengaging mechanism to one or more of the partial (tooth) gear(s) ), which might be used in purpose such as (but not limited to) to form a shape simulating "mouse-like" operations, in which 113 can rotate to (for example but not limited to ) around 150 degrees so that the overall shape of the apparatus (and the support) feels like that of a mouse to index and other fingers, while 115 might be selectively folded or rotate to an "expanded" position (for example but not limited to ) around 120-150 degrees from the folding position so that thumb can reach the controls easily in the "mouse-like" configuration/form of the apparatus.
Detailed Description of Specific Embodiments
[00105] In an embodiment for an apparatus (controller) for user interacting with computer system (such as but not limited to a game controller) comprised of:
[00106] Multiple "axis" of control such as joysticks and buttons/keys and/or touch areas on the surface of the control in a layout normally allows multiple controls being manipulated/operated simultaneously by different fingers, which designed for a certain (specific) holding pose/gesture (such as 2 hands on the handle with thumbs on the joysticks/keypads) which considered the "standard/normal" (by design) gesture (hand/finger pose) for this controller";
[00107] A touch sensitive area (such as but not limited to touch pad, touch screen) that is located on the surface of or articulated to said handheld/wearable device that can be conveniently reached by index finger or middle finger while user's hands/other fingers in corresponding "Normal/Standard" (game controller) operating gesture/ (pose of hand/fingers) ~ such as thumb "on the control"(such as the joy stick)-- and allowing the finger (index finger or middle finger) that perform the finger writing to be in a
"normal'Vcomfortable hand/finger writing position for the finger writing input on the touch sensitive area;
[00108] one example (but not limited to) of said "normal'Vcomfortable finger writing position could be: the location of the touch area is within the range of the index and middle finger when user hold a "normal/standard operation pose" for normal
input/manipulation when not doing text (finger writing) input— the possible locations (of the furthest point of the touching surface) are within the sphere which centered from the knuckle of a index finger(or the point index finger connects to the palm) (normal holding pose) and radius is the length of the index finger. Also it is possible the "plane" of the touch pad/area being ergonometric arranged ~ for example at an angle to the plane of the thumb control (such as joystick), or in a modular design it could be add-on when needed to a convenient location / removed when not needed, or in a foldable/displace-able design it can "stick out/pop up'Vextracted to a convenient location allowing comfortable finger writing;
[00109] "finger writing" recognition modules (it could be for example but not limited to the same as an existing handwriting module) either on board the controller or on (host) computer system connected (such as but not limited to PC, tablet, smart TV or game console);
[00110] So that the said "finger writing" recognition modules process the touch/finger writing events/data from said touch sensitive area (and desirably also take into account other events/info generated by user at the same time such as (but not limited to) those defined as "function keys") and generate (or "translate to")
character(s)/symbol(s)/command(s) as user input (to the downstream system).
[00111] (in one related embodiment) It is desirable key(s)/button(s)/touch sensitive area(s), which can be pressed/held/touched by other fingers when user performing the finger writing can be arranged on the (surface of) input device/controller, desirably arranged on a plane that is (usually) at an angle (such as but not limit to 90 degree) with the plane of the touch pad/area for finger writing, so that other finger(s) (including those used to hold the handheld device) can perform additional input (for example but not limit to :
simultaneously), such as but not limit to holding the defined "function keys" such as shift, Ctrl or Fn key, or perform selection for candidates (character, symbol) right after the finger writing, (and determine the output character/symbol or direction/command), for example if user pressing(holding) additional keys on said controller (such as those could be defined as function keys such as "shift" or "Fn" key) at the time performing the finger writing, the output will be the same as user type on the keyboard and press(holding) the corresponding function key (such as shift or Fn key) on the keyboard.
[00112] In one related embodiment it is desirable additional keys/button (for example but not limit to function keys such as Shift, Ctrl, Fn and etc) are on the same handheld device/wearable device and arranged in places user's other finger can operate these keys/buttons while user performing the finger writing input (such as arranged on the side of the same controller where other fingers or thumb can easily reach while holding the controller so that user can operate with one hand; [00113] In another related embodiment it is desirable additional keys/button (for example but not limit to function keys such as Shift, Ctrl, Fn and etc) are on the same handheld device/wearable device and arranged in places user's other finger can operate these keys/buttons while user performing the finger writing input (such as arranged on the side of the controller(s) where fingers or thumb of the other hand can easily reach while holding the controller so that user can operate with both hands;
[00114] It is desirable the recognition process mentioned above take into account (if any) simultaneous input by other fingers or gestures detected by the
controller/input system to form the end output result (for example if user hold down key(s) on the side of the controller (such as defined as Ctrl or Fn key), the output result is like user hold down these keys and type on the keyboard)
[00115] In another embodiment of an apparatus for user interacting with computer system comprised of:
[00116] A wearable controllers such as a glove like game controller could have controls and the touch area for finger writing input on 2 planes (one for index finger write input, one for thumb joystick like input) which are at angle to each other (such as but not limit to perpendicular), such as shown in Fig 5, (1) is the touch area for index finger for finger writing input and (2) is the touch area or joystick/button(s) for thumb to control direction, just like on a normal game pad/controller. (1) and (2) might be foldable/collapsible when they are not in use, so that they wont obstacle/hinder hand/fingers performing other activity/gestures.
[00117] In an embodiment, a wearable device for tracking user gesture, for example a glove (half glove)-like wearable with sensors (or beacon/makers or trackers) on it, also have a insert-able/modular or collapse-able touch input module (such as a touch pad/screen) that can be reached by user's index finger (optionally in an on demand manner);
[00118] Optionally the glove have other touch/press input areas or buttons so that other fingers can provide input simultaneously,.
[00119] The structure supporting the touch/press input interface could be folded/compressed so that when not needed, it will not interference other hand gesture recognition (such as grab); [00120] "finger writing" recognition modules either on board the glove-like wearable controller or on (host) computer system connected (such as but not limited to PC, tablet, smart TV or game console and etc.);
[00121] So that the said "finger writing" recognition modules process the touch/finger writing events/data from said touch sensitive area (and desirably also take into account other fingers movement/gesture such as those for "function keys") and generate (or "translate to") character(s)/symbol(s)/command(s) as user input (to the downstream system).
[00122] In a related embodiment, it is desirable the finger writing can be interpreted in the context of the gesture by other fingers/palm or hand.
[00123] In a related embodiment other finger's gesture can be use to select candidates of the recognition.
[00124] In another embodiment of an apparatus for user interacting with computer system comprised of:
[00125] A wearable device for tracking user gesture, for example a glove (or half glove)-like wearable with sensors (or beacon/makers or trackers) on it for tracing user's hand movement and/or gesture, also touch input module (such as a touch pad/screen) located on the side of the thumb (same side as the hand back) that can be conveniently reached by user's index finger (or middle finger) of the same hand;
[00126] "finger writing" recognition modules either on board the glove-like wearable controller or on (host) computer system connected (such as but not limited to PC, tablet, smart TV or game console and etc.);
[00127] So that the recognition modules process the touch/finger writing events/data from the touch area (and desirably also take into account other fingers movement/gesture such as those for "function keys") and generate character(s)/symbol(s) as user input (to the downstream system)
[00128] In an embodiment of a apparatus to for character/symbol input from a handheld or wearable interactive device/controller which allows multiple fingers to provide input (to for example but not limited to a computer system/host system such as but limited to game console/TV/setup box/audio system/appliances which the device/controller connected or controlling) conveniently (and possibly even with single-hand) that do not require the help from (user's) vision or voice comprised of: [00129] A compact/limited size(such as but not limited to: stamp-sized, large enough for using finger writing methods to "draw'Vwrite one character, and desirably not too big -such as allowing more than 2 characters— to make it difficult to arrange/port on a controller for an ergonometric location) touch sensitive means/area (such as but not limited to touch pad, touch screen and etc.) that is on the surface of or articulated to said
handheld/wearable device at an (ergonometric) location that can be easily reached by user's finger tip (or top part of the thumb) in a "normal'Vcomfortable hand/finger writing position; (the size of the touch sensitive area is no less than 6mm x 6mm, and maybe not need to be bigger than the size to accommodate a normal finger writing character/symbol);
[00130] It is desirable said touch area might have boundaries/(tactile) hints to indicate the effective touch sensitive area for finger writing input, such as but not limited to: protrusion/recess on the surface or raised/ depressed areas, edges, bumps and etc , it also could be surface (material) with different texture/tactile feeling than surrounding areas, so that user can distinguish the area using finger with tactile feeling alone without the need of visual (confirmation or help/aid to locate) ;
[00131] Means for sending "touch information/data" ("finger writing") collected from the sensor (such as but not limited to touch location/coordinates, events, patterns, force/magnitude (if the pad can provide) and etc) to finger writing recognition module which either onboard the handheld/mobile/wearable device/controller or on (host) computer system connected - if the recognition module is on board this controller the means could be as simple as wired connection, and if the recognition module is considered remote then means for remote connection (such as wired or wireless connection and related protocols) will be needed, and might need to add additional information packet to transmitting the touch events/data (such as by modifying/adding to existing communication protocol);
[00132] Module for "finger writing" recognition (which for example but not limited to be basically same as handwriting recognition module) that process said "touch information/data" ("finger writing");
[00133] So that the said modules for "finger writing" recognition process the touch/finger writing events/data from said touch area (and desirably also take into account other events/info generated by user at the same time such as (but not limited to) those defined as "function keys") and generate (or "translate to") character(s)/symbol(s)/command(s) as user input (to the downstream system);
[00134] It is desirable key(s)/button(s)/touch sensitive area(s), which can be pressed/held/touched by other fingers when user performing the finger writing can be arranged on the (surface of) input device/controller, desirably arranged on a plane that is (usually) at an angle (such as but not limit to 90 degree) with the plane of the touch pad/area for finger writing, so that other finger(s) (including those used to hold the handheld device) can perform additional input, such as but not limit to holding the defined "function keys" such as shift, Ctrl or Fn key, or perform selection for candidates (character, symbol) right after the finger writing, (and determine the output character/symbol or direction/command), for example if user pressing(holding) the shift of Fn key at the time performing the finger writing, the output will be the same as user type on the keyboard and press(holding) the shift or Fn key.
[00135] And it is desirable the recognition result might be combined with other input (if any) from user such as key/button status user holding/pressed or gesture user is making (at the same time user performing the finger writing) to form the result of the output character(s)/symbol(s) or command(s); for example if user pressing(holding) the shift of Fn key at the time performing the finger writing, the output will be the same as user type on the keyboard and press(holding) the shift or Fn key.
[00136] In a related embodiment, it is possible the data/events/information from said touch pad/area is transmitted to said finger writing recognition module in a "realtime or near-realtime" fashion such as but not limited to events/information "packets" about the location/coordinate of the current touch point and timestamp, or the "segment" of the current (continues) curve, or it could be the pattern when touch system determines that the whole character is inputted;
[00137] In a related embodiment, it is desirable to combine other key/button press events or gestures (such as but not limit to those of other fingers) that user performed simultaneously (from the same hand or the other hand), for determining the output character/symbol or direction/command; for example if user pressing(holding) the shift of Fn key (by the fingers on the same hand or the other hand, if applicable such as in case both hands have controllers) at the time performing the finger writing, the output will be the same as user type on the keyboard and press(holding) the shift or Fn key.
[00138] In a related embodiment to all the above, the apparatus might have/expose an interface (which output character/symbol) that is similar or simulate that of a keyboard interface or general HID interface that includes typing/character input, or a COM/stream interface;
[00139] In another embodiment of a apparatus for convenient character input
(and could even with single hand) from a handheld or wearable pointing device/controller such as but not limited to a mouse or 3D mouse, 3D controller which provides
positional/movement input (to the computer system it connects to) without interfering existing functionalities/ operations/navigation comprised of:
[00140] A compact/limited size(such as but not limited to: stamp-sized, large enough for using finger writing methods to "draw'Vwrite one character, and desirably not too big -such as allowing more than 2 characters— to make it difficult to arrange/port on a controller for an ergonometric location) touch sensitive means (such as but not limited to touch pad, touch screen and etc.) that is on the surface of or articulated to said
handheld/wearable pointing device at a "convenient reachable" that can be easily reached by user's (index or middle) finger tip in a "normal'Vcomfortable hand/finger writing position; (the size of the touch sensitive area is no less than 6mm x 6mm, and maybe not need to be bigger than the size to accommodate a normal finger writing character/symbol);
[00141] Such "convenient reachable" location of touch area /means is on (the surface) or articulated to said input device (such as but not limited to mouse, 3D mouse, 3D controller) that can be reached by index finger or middle finger and perform "finger writing" while maintaining a normal operation gesture the device; for example the touch area could be on the "left" or "middle" button of the mouse, or it could be in the area that is about the location of "middle button" or further up, so that left-handers can also operate such touch pad/area with ease;
[00142] "finger writing" recognition module (which could be for example but not limit to the same or similar to handwriting recognition modules already available) on board the controller or "remote" on the (host) computer system connected in which case require transmitting the information/data/events of finger writing from the controller to the host computer system), it is desirable such recognition might take consideration of (if any) simultaneous input by other fingers - such as key/button press from additional keys on the controller such as but not limited to additional keys/function keys on side of the mouse/3D mouse or 3D controller which can be operated by user's other finger at the same time) or gestures detected by the controller/input system to form the end output result (for example if user hold down Ctrl or Fn key, the output result is like user hold down these keys and type on the keyboard)
[00143] So that the said "finger writing" recognition modules process the touch/finger writing events/data from said touch area (and desirably also take into account other events/info generated by user at the same time such as (but not limited to) those defined as "function keys") and generate (or "translate to") character(s)/symbol(s)/command(s) as user input (to the downstream system).
[00144] As the suitable system/method/apparatus here may be embodied in a wide variety of forms, some of which may be quite different from those of the disclosed embodiment. Consequently, the specific structural and functional details disclosed herein are merely representative; yet in that regard, they are deemed to afford the best embodiment for purposes of disclosure and to provide a basis for the claims herein which define the scope of the present invention.
[00145] Discussion of some typical scenarios:
[00146] 1. Game controller: sending messages while playing, or easily input URL/text for browsing/searching
[00147] User take a standard/normal grabbing/operation pose/gesture for the game controller, and while playing the game (such as navigating using thumb), user can easily use index finger to touch the touch sensitive area for finger writing input as described in this invention, and input text message for teammates or for search text box; when browsing the catalog/internet, user can also easily use this finger writing pad to directly input text to the URL or search window without the need for a QWERT keyboard.
[00148] 2. Smart TV: User take a standard grabbing pose for the remote controller, and while watching TV, user can easily use index or thumb to input character on remote controllers equipped with this "finger writing" touch area/pad. The touch/finger writing can be recognized by the recognition system, and provide text or command to downstream system/interface. Optionally the touch system might also recognize swiping, zooming or other navigation command similar to that of the touch screen of a smart phone.
[00149] 3. Mouse:
[00150] While using right hand grabbing the mouse for pointing and clicking, at the same time user can use middle or index finger to input characters or commands on the "finger writing" touch pad on the mouse, and can instantly inserted into text, search box or add quick annotation without hand leaving the mouse.
[00151]
[00152] <1> In one embodiment, the variable geometry hand-based controller/input apparatus comprised of movable parts that can form at least 2
shapes/models/form/positions, one form/shape is the "fully expanded" mode/form that can accommodate user's hand and allow user to manipulate controls easily/ergonometrically (or : without using great efforts), and simultaneously. For example, the surface of a control stick /joy stick might be not on the same plane of some buttons/triggers, when the facing direction of each control is optimized for the finger, for example the "plane" of the surface of control for thumb it is at an angle (such as but not limited to perpendicular) to the palm and for index finger to operate simultaneously, the plane of surfaces for control operated by index finger (such as buttons /triggers) is at an angle to the 1 st plane of surface of control for the thumb (such as but not limited to around 90 degree), such as the situation in Xbox or PS. Actually, so to keep user with the same experience, we can "simulate" these controllers by making the corresponding arrangement of controls for thumb and other fingers with the same or similar direction/position/space between with the gamepad controller, (also make sizes of these controls such as joystick, buttons and their relative spatial position/layout similar to that of a gamepad controller), this is done by adjusting the angel between the 2 planes for surface of control for thumb and other fingers respectively and adjusting control layouts on these planes, so that in the "expanded'Vfunctional form/shape, it have a similar feel/control experience as the gamepad (being simulated);
[00153] There could also be a "low-profile" form in which these 2
planes/surfaces folded/collapsed to a relatively smaller (overall) volume/size/profile (or a more "flat" shape), for example in this mode/form the profile height of the apparatus (the maximum distance of all points on the apparatus to the plane of the palm) is no more than 60% of the "expanded'Vfunctional form. In this way, user can (more easily) using their hands/fingers for other gesture (such as simulating a key press or swipe) or purpose (such as press a button/key on another device);
[00154] The shape/form change/transformation is driven by a driving mechanism (for example but not limited to servo(s) ), and it is desirable such change might be triggered or on-demand;
[00155] In one related embodiment the driving mechanism can drive at least 2 members with planes/surfaces for hand based control (as mentioned above) to transform (such as but not limited to rotate/move/displacement, such as like rotating 90 degrees ) from collapsed/folding position/form to expanded/functional position, and might cause the 2 planes/surfaces to interlock to form a stable controller.
[00156] <2> In one embodiment, the variable geometry hand-based controller/input apparatus comprised of 2 part that moves on different axis(direction), so that 2 surfaces at an angle (such as but not limited to 90 deg) can be formed to accommodate user's thumb and index finger (and other fingers) respectively - user can reach controls on these 2 surfaces easily and comfortably without need to adjust held/grabbing
position/posture/gesture or the grip, so that to allow coordinated/simultaneous
operation/manipulation of the control from thumb and other fingers (such as aiming with thumb and at the same time shooting/pull the trigger with index finger)
[00157] The surface could be arched so that they feels more ergonometric and allow smaller "profile" when collapsed, just like Fig 1.
[00158] In fig.1, one (or more) limited-(distance/rotation)travel or activation mechanism, such as a servo with drive train with partial (tooth) gears can be used to provide "2 (or more) phase" movement from one rotational action from an actuator (such as a servo 105), phase 1 is open one surface for finger controls (such as 113) for the index finger, desirably at multiple (adjustable/customizable) angles, phase 2 is erecting/rotating up the "thumb control" surface, optionally phase 3 could be interlocking them together (powered or manually).
[00159] <3> In one embodiment, an ergonometric variable geometry hand- based controller/input apparatus comprised of: a 1st member with surface for hand/finger control , and 2 articulated member that can rotate along 2 different axis, one of the member might be ergonometric shaped to fit in to of the palm and desirable can be optionally attached to the hand (palm) and move together with the hand without relative movement, so that to said 1st member to rotate "against" the palm via this member thus allow the apparatus to transform to a different form/shape ("operational state") with different angles rotated, for purpose such as but not limited to: simulating the physical shape/surface/interface/grasp of different tools/handles of different sizes/shapes; Another articulated member with surface for hand/finger control ("2nd control surface member") could also rotate against said 1st member, and by itself can also have controls on it, so that to provide 2 surfaces with hand/finger control on them at an angle with each other (such as but not limited to 90 degrees), so that user's thumb and other fingers can operate these controls simultaneously (could be similar to the way user operate a gamepad); Said "2nd control surface member" can rotate to an angle that is parallel (or close to parallel such as within 20 degrees of parallel) so that it can be "collapsed'Vfolded and allow the 1 st member to rotate and ensure a low profile when the device is in a "low-profile" mode/form and with minimum
interference/blocking of hand/finger movement for other purposes/operations such as but not limit to gesture input/control or touching other buttons of another control.
[00160] There could be one (or more) actuator on the said control surface or articulated members to driven such shape change according to control. One design is to use a servo at the bottom part of the control surface, and it connected to a transmission shaft which have partial (tooth) gear (gear with only a part of the whole arc have tooth, other part are smooth and not driving the movement of the conterpart/meshed gear) on it, one such gear drives the rotation against the member that "supports/touches" the palm(means it touches the palm and with no relative movement with the palm), another partial (tooth) bevel gear drives 2nd control surface member, when the 1st control surface is in a "expanded" mode / angle (greater than 60 degrees) to the palm that allow enough space to rotate the 2nd control surface member "up" (into functional position);
[00161] In a related embodiment it is desirable there could be a latch/locking mechanism between the (moving) members which could be driven by an actuator or by hand to "lock" the relative position (such as when 2nd surface is fully expanded) between 2 or more members so that the control surface have a more "solid" feel when operated by the user. [00162] There could be additional state for simulating a mouse where the 2nd surface is not expanded and the 1st control surface is rotated further (more than 90 degrees, close to 120 or 150 degrees) so that provide support to palm when use it as palm face down gestures like a mouse. Also, there could be position or motion tracking means such as optical mouse chip and lenses on the apparatus that is facing to the "desk" surface (or: the direction of what the palm is facing) when this "mouse mode" is used and provide movement
(coordinates or increments) info accurately.
[00163] The embodiments of the invention described above are intended to be merely exemplary; numerous variations and modifications will be apparent to those skilled in the art. All such variations and modifications are intended to be within the scope of the present invention as defined in any appended claims.

Claims

Claims What is claimed is:
1. A method to allow convenient and possibly single-hand character/symbol input from a or "a pair of handheld or wearable interactive device/controller (such as but not limit to a game controller) which allows multiple fingers to provide input (to the computer system it connects to) simultaneously without interfering existing functionalities/finger
operations/game play (i.e. the ability to operate the joystick with the thumb, and other buttons as usual) includes:
Providing user a touch sensitive means (such as but not limited to touch pad, touch screen and etc.) that is:
compact in size such as but not limited to: stamp-sized, or large enough for using finger writing methods to "draw'Vwrite one character, and desirably not too big— such as allowing more than 2 characters— in area so that to make it difficult to arrange/port on a controller for an ergonometric location;
on the surface of or articulated to said handheld/wearable device and the touch area have a surface facing direction for "normal'Vcomfortable hand/finger writing and at a "convenient reachable" location that can be easily reached by user's finger tip in a "normal'Vcomfortable hand/finger writing position;
Said "convenient reachable" location of touch area /means is on the surface of or articulated to said input device such as but not limited to the game controller at a location that can be reached by index finger or middle finger and perform "finger writing" while maintaining a normal operation gesture for other controls/buttons on the input device/controller - in another words, the location of the touch area is within the range of the index and middle finger when user holding a "normal operation status" for normal input/manipulation when not doing text (finger writing) input ~ the possible locations (of the furthest point of the touching surface) are within the sphere which centered from the knuckle of a index finger(or the point index finger connects to the palm) (normal holding pose) and radius is the length of the index finger,
Performing "finger writing" recognition (by recognition modules on board the controller or transmitted the information/data/events of finger writing from the controller to the (host) computer system connected, such as but not limited to PC, tablet, smart TV or game console and perform recognition there), it is desirable such recognition might take consideration of (if any) simultaneous input by other fingers or gestures detected by the controller/input system to form the end output result (for example if user hold down Ctrl or Fn key, the output result is like user hold down these keys and type on the keyboard);
Outputting the result of the recognition (such as character(s)/symbols or commands) to a down stream system/interface (such as but not limited to a text or keyboard receiving interface, or GUI elements such as a text/URL/search input box, F E(input method editor) window, and etc. ).
2. A method according to claim 1, further includes prompting user such as visually or audibly for candidates of recognition, or auto-completion;.
3. A method according to claim 1, wherein said touch area pad might be modular and can be added/removed from the controller.
4. A method according to claim 1, said touch area pad is foldable or collapsible or can be displaced so that when user do not need to finger writing input, the touch area/pad can be "folded/collapsed" or removed from the "operating position" to prevent it from
obstacle/hindering normal operation of fingers to the controls (such as game pad
buttons/joysticks) .
5. An apparatus (controller) for user interacting with computer system (such as but not limited to a game controller) comprised of:
Multiple "axis" of control—such as joysticks and buttons/keys ~ and/or touch areas on the surface of the control in a layout normally allows multiple controls being
manipulated/operated simultaneously by different fingers, which designed for a certain (specific/" standard'V'normal") holding pose/gesture ~ for example, a "standard/normal" (by design) gesture (hand/finger pose) for a game controller is 2 hands on the handle with thumbs on the joysticks/keypads;
A touch sensitive area (such as but not limited to touch pad, touch screen) that is on the surface of or articulated to said handheld/wearable device that can be conveniently reached by index finger or middle finger while user's hands/other fingers remains in said
corresponding "Normal/Standard" (game controller) operating gesture/ (pose of
hand/fingers) of the device ~ such as thumb "on the control"(like the joy stick) in case of a game controller— and allowing the finger (index finger or middle finger) that perform the finger writing to be in a "normal'Vcomfortable hand/finger writing position for the finger writing input on the touch sensitive area, the location of the touch area is within the range of the index and middle finger when user holding said "normal/standard" operating gesture/pose for normal input/manipulation ~ for example the possible locations (of the furthest point of the touching surface) are within the sphere which centered from the knuckle of a index finger(or the point index finger connects to the palm) (normal holding pose) and radius is the length of the index finger, or it is also possible the "plane" of the touch pad/area being ergonometric arranged for example have an angle to the plane of the thumb control (such as joystick), or in a modular design it could be add-on when needed to a convenient location / removed when not needed, or in a foldable/displace-able design it can "stick out/pop up'Vextracted to a convenient location allowing comfortable finger writing;
"finger writing" recognition modules (it could be for example but not limited to the same as an existing handwriting module) either on board the controller or on (host) computer system connected (such as but not limited to PC, tablet, smart TV or game console);
So that the said "finger writing" recognition modules process the touch/finger writing events/data from said touch sensitive area (and desirably also take into account other events/info generated by user at the same time such as (but not limited to) those defined as "function keys") and generate (or "translate to") character(s)/symbol(s)/command(s) as user input (to the downstream system).
6. An apparatus according to claim 5, key(s)/button(s)/touch sensitive area(s), which can be pressed/held/touched by other fingers when user performing the finger writing can be arranged on the (surface of) input device/controller, desirably arranged on a plane that is (usually) at an angle (such as but not limit to 90 degree) with the plane of the touch pad/area for finger writing, so that other finger(s) (including those used to hold the handheld device) can perform additional input (for example but not limit to : simultaneously), such as but not limit to holding the defined "function keys" such as shift, Ctrl or Fn key, or perform selection for candidates (character, symbol) right after the finger writing, (and determine the output character/symbol or direction/command), for example if user pressing(holding) additional keys on said controller (such as those could be defined as function keys such as "shift" or "Fn" key) at the time performing the finger writing, the output will be the same as user type on the keyboard and press(holding) the corresponding function key (such as shift or Fn key) on the keyboard.
7. An apparatus according to claim 5 additional keys/button (for example but not limit to function keys such as Shift, Ctrl, Fn and etc) are on the same handheld device/wearable device and arranged in places user's other finger can operate these keys/buttons while user performing the finger writing input (such as arranged on the side of the same controller where other fingers or thumb can easily reach while holding the controller so that user can operate with one hand;
8. An apparatus according to claim 5 wherein said recognition process take into account (if any) simultaneous input by other fingers or gestures detected by the controller/input system to form the end output result (for example if user hold down key(s) on the side of the controller (such as defined as Ctrl or Fn key), the output result is like user hold down these keys and type on the keyboard):
9. An apparatus for user interacting with computer system comprised of:
A wearable device for tracking user gesture, for example a glove (or half glove)-like wearable with sensors, beacon, makers or trackers on it for tracing user's hand movement and/or gesture, also touch input module (such as a touch pad/screen) located on the side of the thumb (same side as the hand back) that can be conveniently reached by user's index finger (or middle finger) of the same hand;
"finger writing" recognition modules either on board the glove-like wearable controller or on (host) computer system connected (such as but not limited to PC, tablet, smart TV or game console and etc.);
So that the recognition modules process the touch/finger writing events/data from the touch area (and desirably also take into account other fingers movement/gesture such as those for "function keys") and generate character(s)/symbol(s) as user input (to the downstream system)
10. An apparatus according to claim 9, finger writing can be interpreted in the context of the gesture by other fingers/palm or hand.
11. An apparatus according to claim 9, wherein other finger's gesture can be use to select candidates of the recognition.
12. An apparatus of Variable Geometry hand-based control comprised of:
At least 2 articulated or touched members that can rotate/move/displacement against each other and at least one of the members can fit into the operation range of a user's hand/fingers (such as but not limited to: the size fits into user's palm), and one or more controlled actuator/powered driving mechanism on the apparatus and operationally connected/coupled to at least one(or 2) members that is able to rotate/move/displacement at least one
"active/movable" member rendering the geometry form/shape change from to an other following control from a controller for form/shape control (which desirably could , either coordinates with or controlled by or can processing request from, appropriate user interface(s) so that the current form of the apparatus is consistent with the current (intended) user interface for proper interaction requirements/preferences which could be different for different apps/software that needs interaction/input from user);
there is a least one "surface for hand/finger control" (with hand/finger operated controls on it) located on least one of said "active/movable" member(s) which is(are) operationally connected to the driving mechanism and can move/rotate/displace (relative to other member(s));
In at least one of the geometry form/shapes of said hand-based control apparatus allows user to use hand/finger to operate the controls on the "surface for hand/finger control"— such as but not limited to: performing control of machine connected to the apparatus, performing input/interaction such as but not limit to touch input, pressing keys/buttons, manipulating joystick and etc.;
In at least one another geometry form/shapes of said hand-based control apparatus allowing user to performing hand-based operation(s) that is in a different manner with the previous one, for example but not limited to using gesture input (without any buttons or touch inputs) or at least using a group of controls that is not entirely the same as the previous one (which means the hand-operable controls in this form could be a subset or supper set of that in the previous ones or it could include different controls/input means);
So that by switching/changing to different geometry forms/shapes, the apparatus allows user to perform control/interaction with a computer implemented system (that
expects/requires user interaction) in different manner/form without the need to change to another controller, and might also be able to simulate the physical experience of different kind of controls.:
13. An apparatus according to claim 12, wherein at least one of the folding axis of a moveable member is (substantially) parallel to that of the average folding axis (vector) of the fingers of a user's hand excepting the thumb.
PCT/IB2016/000998 2015-07-13 2016-07-13 Apparatus and method for hybrid type of input of buttons/keys and "finger writing" and low profile/variable geometry hand-based controller WO2017009707A1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
PCT/IB2016/000998 WO2017009707A1 (en) 2015-07-13 2016-07-13 Apparatus and method for hybrid type of input of buttons/keys and "finger writing" and low profile/variable geometry hand-based controller
EP16823943.2A EP3323036A4 (en) 2015-07-13 2016-07-13 Apparatus and method for hybrid type of input of buttons/keys and "finger writing" and low profile/variable geometry hand-based controller
CN201680053129.6A CN108027705A (en) 2015-07-13 2016-07-13 The apparatus and method inputted for buttons/keys and " finger writing " mixed type and low gabarit/geometry-variable controller based on hand

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US201562191528P 2015-07-13 2015-07-13
US62/191,528 2015-07-13
US201562210403P 2015-08-26 2015-08-26
US62/210,403 2015-08-26
PCT/IB2016/000998 WO2017009707A1 (en) 2015-07-13 2016-07-13 Apparatus and method for hybrid type of input of buttons/keys and "finger writing" and low profile/variable geometry hand-based controller

Publications (1)

Publication Number Publication Date
WO2017009707A1 true WO2017009707A1 (en) 2017-01-19

Family

ID=57757964

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2016/000998 WO2017009707A1 (en) 2015-07-13 2016-07-13 Apparatus and method for hybrid type of input of buttons/keys and "finger writing" and low profile/variable geometry hand-based controller

Country Status (3)

Country Link
EP (1) EP3323036A4 (en)
CN (1) CN108027705A (en)
WO (1) WO2017009707A1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110730940A (en) * 2017-04-13 2020-01-24 脸谱科技有限责任公司 Handheld controller using segmented capacitive touch trigger
CN112972909A (en) * 2019-12-13 2021-06-18 医科达（北京）医疗器械有限公司 Controller for radiotherapy equipment
CN114648756A (en) * 2022-05-24 2022-06-21 之江实验室 Book character recognition and reading method and system based on pointing vector
US11991222B1 (en) 2023-05-02 2024-05-21 Meta Platforms Technologies, Llc Persistent call control user interface element in an artificial reality environment

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109965482A (en) * 2018-08-06 2019-07-05 深圳市奋达科技股份有限公司 Automatic curling iron
CN109782957B (en) * 2019-01-24 2023-03-14 吴宏琴 False touch prevention method based on game scene, mobile terminal and storage medium
CN111459265B (en) * 2020-03-02 2023-10-13 杭州嘉澜创新科技有限公司 Interactive device, operation method thereof and computer readable storage medium
CN112109644B (en) * 2020-09-11 2022-03-04 中国第一汽车股份有限公司 Vehicle control method, device, equipment and storage medium
CN115237251A (en) * 2020-11-10 2022-10-25 深圳市瑞立视多媒体科技有限公司 Method for screwing screw in virtual reality environment
US11487365B2 (en) * 2021-02-03 2022-11-01 Christian Ryan Umali Leonardo Electronic apparatus for repositioning and transitioning among input devices
CN115337629A (en) * 2021-05-14 2022-11-15 深圳市金洋汇创科技有限公司 Game machine controller and game machine
WO2023178564A1 (en) * 2022-03-23 2023-09-28 黄翔宇 Touchpad game controller

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102043534A (en) * 2010-10-13 2011-05-04 苏州佳世达电通有限公司 Electronic device control method and electronic device
US20120293410A1 (en) * 2011-05-18 2012-11-22 Ian Bell Flexible Input Device Worn on a Finger
CN103034438A (en) * 2012-11-26 2013-04-10 上海应用技术学院 Method for realizing multi-screen multi-point touch control browser
CN203301578U (en) * 2013-06-06 2013-11-20 陈志春洋 Cellphone with auxiliary touch controller
CN104393897A (en) * 2014-12-07 2015-03-04 王�义 Intelligent Bluetooth wearable equipment worn on thumb

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1248972A2 (en) * 2000-01-20 2002-10-16 Oliver Kauk Data input device
CN201084119Y (en) * 2007-07-30 2008-07-09 张运宏 Gloves type mouse
CN102667584A (en) * 2009-11-16 2012-09-12 金时焕 Portable display apparatus
CN201820210U (en) * 2010-10-27 2011-05-04 胡枝清 Detachable touch pad and wireless mouse in notebook computer
JP5379176B2 (en) * 2011-01-25 2013-12-25 株式会社ソニー・コンピュータエンタテインメント Portable electronic devices
EP3734407A1 (en) * 2011-02-10 2020-11-04 Samsung Electronics Co., Ltd. Portable device comprising a touch-screen display, and method for controlling same
CN204206404U (en) * 2014-10-28 2015-03-11 刘志伟 A kind of wearable communication headset

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102043534A (en) * 2010-10-13 2011-05-04 苏州佳世达电通有限公司 Electronic device control method and electronic device
US20120293410A1 (en) * 2011-05-18 2012-11-22 Ian Bell Flexible Input Device Worn on a Finger
CN103034438A (en) * 2012-11-26 2013-04-10 上海应用技术学院 Method for realizing multi-screen multi-point touch control browser
CN203301578U (en) * 2013-06-06 2013-11-20 陈志春洋 Cellphone with auxiliary touch controller
CN104393897A (en) * 2014-12-07 2015-03-04 王�义 Intelligent Bluetooth wearable equipment worn on thumb

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3323036A4 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110730940A (en) * 2017-04-13 2020-01-24 脸谱科技有限责任公司 Handheld controller using segmented capacitive touch trigger
CN112972909A (en) * 2019-12-13 2021-06-18 医科达（北京）医疗器械有限公司 Controller for radiotherapy equipment
CN114648756A (en) * 2022-05-24 2022-06-21 之江实验室 Book character recognition and reading method and system based on pointing vector
US11991222B1 (en) 2023-05-02 2024-05-21 Meta Platforms Technologies, Llc Persistent call control user interface element in an artificial reality environment

Also Published As

Publication number Publication date
EP3323036A4 (en) 2019-05-29
EP3323036A1 (en) 2018-05-23
CN108027705A (en) 2018-05-11

Similar Documents

Publication Publication Date Title
EP3323036A1 (en) Apparatus and method for hybrid type of input of buttons/keys and &#34;finger writing&#34; and low profile/variable geometry hand-based controller
US20210018993A1 (en) Computer mouse
US8199107B2 (en) Input interface device with transformable form factor
US7161579B2 (en) Hand-held computer interactive device
US10705619B2 (en) System and method for gesture based data and command input via a wearable device
EP3234742A2 (en) Methods and apparatus for high intuitive human-computer interface
US20200310561A1 (en) Input device for use in 2d and 3d environments
WO2015113503A1 (en) Ring-type wireless finger controller, control method and control system
US20080129682A1 (en) 3-D Computer Input Device and Method
US20090322680A1 (en) Radio frequency pointing device
WO2020087999A1 (en) Hand action capturing device having force feedback
Toh et al. Dexterous telemanipulation with a multi-touch interface
US11861064B2 (en) Wearable data input device and operating method
CN111770815A (en) Object control method and object control device
Bai et al. Asymmetric Bimanual Interaction for Mobile Virtual Reality.
TW200807284A (en) Programmable touch system
US20200192477A1 (en) Modular hand-mounted input device
CN204557377U (en) Electronics input gloves
CN107817911B (en) Terminal control method and control equipment thereof
KR20090085821A (en) Interface device, games using the same and method for controlling contents
WO2014072786A2 (en) 3d hand-gesture to convey cartesian space arrangement plus all six degrees of freedom: three rotations and three translations along the cartesian axes
US20130027307A1 (en) Human-machine interface apparatus and operating method thereof
US20230116966A1 (en) A dual peripheral device
WO2015174307A1 (en) Input assistance device
CN116449963A (en) Virtual reality interaction method and device based on VR (virtual reality) head-mounted equipment

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16823943

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2016823943

Country of ref document: EP