US20180067543A1 - 3D Haptics For Interactive Computer Systems - Google Patents
3D Haptics For Interactive Computer Systems Download PDFInfo
- Publication number
- US20180067543A1 US20180067543A1 US15/255,866 US201615255866A US2018067543A1 US 20180067543 A1 US20180067543 A1 US 20180067543A1 US 201615255866 A US201615255866 A US 201615255866A US 2018067543 A1 US2018067543 A1 US 2018067543A1
- Authority
- US
- United States
- Prior art keywords
- contact
- contact device
- user
- actuator
- controller device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/01—Indexing scheme relating to G06F3/01
- G06F2203/013—Force feedback applied to a game
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/01—Indexing scheme relating to G06F3/01
- G06F2203/015—Force feedback applied to a joystick
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/041—Indexing scheme relating to G06F3/041 - G06F3/045
- G06F2203/04101—2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/041—Indexing scheme relating to G06F3/041 - G06F3/045
- G06F2203/04105—Pressure sensors for measuring the pressure or force exerted on the touch surface without providing the touch position
Definitions
- the present disclosure relates to providing haptic feedback to a user of a controller device.
- Interactive computer systems implementing a virtual environment may include, but are not limited to, virtual reality (VR) systems, augmented reality systems, gaming systems, etc.
- VR systems provide a visual and audio representation of a three dimensional (3D) VR environment to a user.
- the VR environment corresponds to a gaming environment.
- a VR system may provide the visual representation of the VR environment using a display, a headset incorporating a display, etc.
- Controller devices include, but are not limited to, handheld (i.e., remote) controllers, gloves or other wearable devices, etc. configured to communicate with the VR system.
- the controller devices may provide indications of respective positions (e.g., 6 degree-of-freedom, or 6-DOF, positions) of the controller devices to the VR system.
- the position indications may further indicate positions, orientations, movements, etc. of the respective hands holding the controller devices.
- the VR system implements optical tracking to determine positions and movements of the controller devices. In this manner, the user is able to interact with (e.g., provide inputs to) the VR environment.
- the controller devices may also include other input mechanisms, such as buttons, touchpads, trackpads, etc. for receiving user inputs and causing further interaction with the VR environment.
- a controller device for a virtual environment includes a handle and a contact device having a substantially planar surface. A position of the contact device relative to the handle is adjustable. An actuator module is arranged to adjust the position of the contact device relative to the handle. A control module in communication with the virtual environment selectively controls the actuator module to adjust the position of the contact device in response to data received from the virtual environment. The data includes an indication of an interaction between a user and an object represented within the virtual environment.
- a method for operating a controller device for a virtual environment includes displaying an object within the virtual environment, generating data including an indication of interaction between a user and a surface of the object, providing the data to an actuator of the controller device, and using the actuator adjusting a position of a contact device relative to a handle of the controller device based on the data.
- FIG. 1 is an example virtual reality system according to the principles of the present disclosure.
- FIG. 2 is a simplified virtual reality system including a controller device shown in more detail according to the principles of the present disclosure.
- FIG. 3A is a side view of a first example controller device according to the principles of the present disclosure.
- FIG. 3B is a plan view of the first example controller device according to the principles of the present disclosure.
- FIGS. 4A and 4B are a first example contact device and actuator module according to the principles of the present disclosure.
- FIGS. 5A and 5B illustrate example operation of the first example controller device according to the principles of the present disclosure.
- FIG. 6A is a side view of a second example controller device according to the principles of the present disclosure.
- FIG. 6B is a plan view of the second example controller device according to the principles of the present disclosure.
- FIG. 7 is a second example contact device and actuator module according to the principles of the present disclosure.
- FIG. 8 illustrates example operation of the second example controller device according to the principles of the present disclosure.
- FIG. 9 is an example method for providing feedback to a user of a virtual reality system according to the principles of the present disclosure.
- Some controller devices for virtual reality (VR) systems implement one or more mechanisms for providing feedback to the user.
- the feedback may be indicative of various features of a VR environment (e.g. proximity to objects, such as walls or other structures, the occurrence of an event within a certain range of the user, etc.) and/or responsive to user behavior within the VR environment (e.g., firing a virtual weapon, interacting with an object, etc.).
- Example feedback includes vibration, audio indicators (e.g., beeps), etc.
- the feedback provided to users via handheld or wearable controller devices may not accurately represent the interaction of the user with the VR environment.
- feedback such as vibration may not accurately convey haptic interaction with objects in the VR environment to the user.
- a controller device includes a contact device (e.g., a platform, pad, etc.) with a contact surface arranged to receive a fingertip of the user.
- the controller device is configured to adjust the contact surface of the contact device according to interaction between the user and the VR environment to provide haptic feedback (e.g., cutaneous and/or kinesthetic feedback).
- haptic feedback e.g., cutaneous and/or kinesthetic feedback.
- the controller device adjusts an orientation of the contact surface in response to the user touching an object or surface in the VR environment.
- the controller device adjusts a texture of the contact surface.
- VR systems Although described with respect to VR systems, the principles of the present disclosure may also be included in other interactive computer systems implementing a virtual environment, including, but not limited to, augmented reality systems and gaming systems using a display such as a pc or laptop monitor, a gaming console using a television, etc. Accordingly, as used herein, the term “virtual reality system” may be used to indicate any interactive computer system implementing a virtual environment.
- FIG. 1 shows an example VR system 100 .
- the VR system 100 includes a host device (e.g., a computer, gaming console, etc.) 104 and one or more peripheral devices, such as a controller device 108 according to the principles of the present disclosure and a display 112 .
- the VR system 100 may include a tracking system 116 (e.g., an optical tracking system) configured to track movements of a user and/or the controller device 108 .
- the VR system 100 may include other inputs 120 (e.g., a keyboard, mouse, wireless inputs, voice input mechanism such as a microphone, etc.) and other outputs 124 (e.g., speakers, secondary displays, wireless outputs, etc.).
- the host device 104 includes one or more interfaces 128 for communicating with the peripheral devices.
- the interfaces 128 may include, but are not limited to, input/output (I/O) interfaces and adapters (e.g., universal serial bus, or USB), and wired or wireless communication interfaces (WiFi, Bluetooth, cellular, wired Ethernet, etc.) for communicating with a network 136 .
- the interfaces 128 allow data to be transmitted between the peripheral devices and/or the network 132 and a control module 136 .
- the control module 136 controls processing of data related to operation of the VR system 100 .
- the control module 136 may correspond to one or more processors configured to execute an operating system of the host device 104 and one or more programs stored in disk storage 140 , memory 144 , a cloud computing system (e.g., via network 132 ), etc. to implement a VR environment.
- the disk storage 140 may correspond to a hard disk drive (HDD), solid state drive (SSD), a removable media drive such as a CD or DVD ROM drive, flash memory, etc.
- the memory 144 may correspond to system memory and may include, but it is not limited to, volatile and/or non-volatile semiconductor memory.
- FIG. 2 shows a simplified VR system 200 including only a host device 204 , controller device 208 , display 212 , and tracking system 216 .
- the controller device 208 includes a control module 220 , a contact device 224 , input sensors 228 , and output actuators 232 configured to implement the systems and methods according to the principles of the present disclosure.
- the controller device 208 includes communication interfaces (e.g., wired and/or wireless communication interfaces) 236 for communicating data between the controller device 208 and the host device 204 .
- the controller device 208 may include other inputs 240 and other outputs 244 .
- the other inputs 240 may include, but are not limited to, a microphone for receive voice inputs, buttons, a touchpad or touchscreen interface (e.g., for providing an indication of a relative position of a fingertip on the contact device 224 ), etc.
- the other outputs 244 may include, but are not limited to, audio outputs, an integrated display, LEDs, other feedback mechanisms (e.g., vibration), etc.
- the control module 220 controls the operation of the controller device 208 based on data received from the host device 204 , contact device 224 , and inputs 240 , and provides data indicative of user interaction with the controller device 204 to the host device 204 as described below in more detail.
- the host device 204 provides interaction data indicative of user interaction with the VR environment to the control module 22 .
- the user interacts with the VR environment (e.g., as presented to the user via the display 212 ) using the controller device 208 .
- the tracking system 216 tracks user movement by monitoring movement of the controller device 208 and provides tracking data indicative of the user movement to the host device 204 .
- the host device 204 generates and outputs the interaction data based on the tracking data received from the tracking system and, in some examples, further based on user input data received from the controller device 208 (e.g., user input data generated in response to user interaction with contact device 224 , inputs 240 , etc.).
- the control module 220 controls the contact device 224 based on the interaction data.
- the interaction data may include characteristics of the interaction between the user and the VR environment. The characteristics may correspond to contact characteristics associated with contact between the user (e.g., a fingertip of the user in the VR environment) and an object in the VR environment, including, but not limited to, surface contour, shape, and texture of the object, as well as an amount of force exerted on and/or by the object.
- the control module 220 controls the output actuators 232 to physically adjust the contact device 224 so that contact between the contact device 224 and an actual fingertip of the user reflects the contact between the fingertip of the user and the object in the VR environment.
- input sensors 228 (which may include force or finger position sensors) provide additional feedback indicative of contact between the user and the contact device 224 .
- the controller device 300 includes a handle 304 , an actuator module 308 arranged on a first end of the handle 304 , and a contact device 312 mounted on the actuator module 308 .
- the contact device 308 is arranged to receive a fingertip 316 of a user.
- the contact device 308 as shown corresponds to a substantially circular platform, the contact device 308 may have any suitable shape in other examples (e.g., rectangular, elliptical, etc.).
- the controller device 300 may include one or more tracking elements 320 (e.g., retroreflective spheres) detectable by the tracking system 216 arranged to facilitate tracking of the movement of the controller device 300 in 3D space.
- tracking elements 320 e.g., retroreflective spheres
- Other examples may implement other suitable systems and methods for tracking movement of the controller device 300 .
- components such as the control module 220 , the actuator module 308 , etc. may be located within a housing of the handle 304 .
- the actuator module 308 selectively physically adjusts the contact device 312 based on interaction data received from the host device 204 as described above with respect to FIG. 2 .
- the actuator module 308 may include servos or other suitable mechanisms in communication with respective actuator arms 324 .
- the actuator module 308 may be configured to independently actuate (e.g., extend and retract) the actuator arms 324 to adjust the position of (e.g., tilt to adjust yaw and pitch, raise, lower, etc.) the contact device 312 relative to the handle 304 (and, therefore, relative to the fingertip 316 ).
- the actuator module 308 adjusts the position of the contact device 312 accordingly.
- the visual representation of the fingertip of the user within the VR environment may also be adjusted to indicate the contact (e.g., by tilting or extruding the finger).
- the contact device 400 includes substantially planar upper and lower platforms 408 and 412 .
- the upper platform 408 may have a concave upper surface 416 .
- the platform 408 is formed from acetal or other plastic material.
- a force sensor e.g., a force transducer
- 420 is arranged between the upper and lower platforms 408 and 412 to detect an amount of force applied to the contact device 400 by a user.
- the actuator module 404 includes servo motors 424 configured to adjust the contact device 400 via respective control arms 428 and actuator arms 432 (e.g., responsive to commands from the control module 220 ). As shown, the actuator module 404 includes three servo motors 424 , control arms 428 , and actuator arms 432 to provide three degrees of freedom movement of the contact device 400 .
- the control arms 428 are connected to the actuator arms 432 via respective revolute joints 436 and the actuator arms 432 are connected to the contact device 400 via ball-and-socket spherical joints 440 .
- the force sensor 420 comprises a force sensing resistor (FSR) material and two internal electrodes for detecting compression between the upper and lower platforms 408 and 412 in response to force applied to the upper platform 408 by the user.
- FSR force sensing resistor
- the upper platform 408 is connected to the lower platform 412 via adjustable screws or other fasteners 444 at respective ends of radial arms 448 around an outer perimeter of the upper platform 408 .
- the arms 444 may be formed in the material of the upper platform 408 to allow downward movement of an inner portion of the upper platform 408 in response to force applied by the user.
- the arms 444 may be configured to bias the upper platform 408 downward against the force sensor 420 even without contact from a user. Accordingly, a nominal voltage may be generated by the force sensor 420 with no user contact, while the voltage may increase proportionately to an amount of force applied to the upper platform 408 by the user. The voltage is provided as feedback indicative of the amount of force applied by the user (e.g., to the control module 220 ).
- FIG. 5A the contact device 500 is shown in a plurality of example positions relative to a fingertip 504 of a user. As shown, the positions of the contact device 500 correspond to a contact point between a virtual fingertip of the user and a surface of an example object 508 in a VR environment.
- the virtual fingertip of the user e.g., as rendered and provided to the user via the display 212
- the virtual fingertip moves relative to the object 508 and contacts different points along the surface of the object 508 .
- the contact device 500 is adjusted (e.g., tilted) as movement of the controller device 300 causes the virtual fingertip to make initial contact with the object 508 , contact different portions of the object 508 , discontinue contact with the object 508 , etc. For example, as the virtual fingertip “slides” along the surface of the object 508 , the contact device 500 is tilted accordingly (e.g. relative to the controller device 300 , a mounting portion such as actuator module 512 , etc.). For example only, the contact device 500 is adjusted such that the contact device 500 is substantially perpendicular to a surface normal 516 of the object 508 at a contact point between the virtual fingertip and the object 508 .
- the contact device 500 is shown in a plurality of positions (e.g., height positions) as a virtual fingertip of a user contacts a surface 520 in the VR environment.
- the virtual fingertip is not in contact with the surface 520 and therefore the contact device 500 is shown in a fully retracted position.
- the virtual fingertip is in contact with the surface 520 .
- the contact device 500 may be extended in response to movement that would otherwise cause the virtual fingertip to penetrate the surface 520 within the VR environment. For example, at 532 , the contact device 500 is extended to apply upward force against the fingertip 504 of the user. In other words, as the controller device 300 is moved downward, the fingertip 504 is nonetheless maintained in a same position due to the opposing, upward movement of the contact device 500 . In this manner, the contact device 500 indicates to the user that the movement of the controller device 300 is causing the virtual fingertip to attempt to penetrate the surface 520 .
- the contact device 500 is extended still further as shown at 536 .
- the position of the contact device 500 as shown at 536 may correspond to a fully extended position. Accordingly, further downward movement of the controller device 300 allows corresponding movement of the fingertip 504 , resulting in penetration of the surface 520 as shown at 540 .
- a position of the virtual fingertip (and a virtual hand, which corresponds to a position of the controller device 300 ) as rendered in the VR environment may be decoupled from further movement of the controller device 300 to compensate for penetration of the surface 520 .
- the contact device 500 may be maintained in a partially extended position (e.g., 50% or 75% of a fully extended position) while maintaining presentation of the virtual fingertip on the surface 520 . Accordingly, if penetration of the surface 520 by the user is inadvertent, the user is nonetheless able to interact with the surface 520 .
- the controller device 600 includes a handle 604 , an actuator module 608 arranged on a first end of the handle 604 , and a contact device 612 mounted on the actuator module 308 .
- the contact device 608 is arranged to receive a fingertip 616 of a user.
- the contact device 608 as shown corresponds to a substantially circular platform, the contact device 608 may have any suitable shape in other examples (e.g., rectangular, elliptical, etc.).
- the controller device 600 may include one or more tracking elements 620 (e.g., retroreflective spheres) detectable by the tracking system 216 arranged to facilitate tracking of the movement of the controller device 600 .
- tracking elements 620 e.g., retroreflective spheres
- components such as the control module 220 , the actuator module 608 , etc. may be located within a housing of the handle 604 .
- the actuator module 608 selectively adjusts the contact device 612 based on interaction data received from the host device 204 as described above with respect to FIG. 2 .
- the contact device 612 includes a tactile array 624 (e.g., as shown, a 4 ⁇ 4 array) of adjustable pins 628 .
- the actuator module 608 includes servos and/or other suitable mechanisms configured to independently actuate respective ones of the pins 628 .
- the actuator module 608 independently actuates (e.g., extends and retracts) the pins 628 to adjust the position of (e.g., tilt, raise, lower, etc.) the tactile array 624 relative to the handle 604 (and, therefore, relative to the fingertip 616 ).
- the actuator module 608 adjusts respective positions of the pins 628 of the tactile array 624 accordingly.
- the contact device 700 includes a substantially planar upper platform 708 including a 4 x 4 tactile array 712 of sixteen pins 716 .
- the tactile array 712 has a 4 ⁇ 4 configuration, other examples may use a different configuration (e.g., 3 ⁇ 4, 5 ⁇ 5, 4 ⁇ 6, etc.).
- the actuator module 704 is configured to independently actuate (e.g., extend and retract) each of the pins 716 based on interaction with an object in the VR environment.
- the actuator module 704 includes a plurality of linear actuator systems 720 in communication with respective ones of the pins 716 , although only a single linear actuator system 720 is shown for simplicity.
- the linear actuator system 720 has a rack and pinion configuration.
- Each of the linear actuator systems 720 includes a servo motor 724 configured to rotate a first spur or pinion gear 728 .
- Rotation of the first spur gear 728 causes linear actuation of rack gear 732 , which in turn causes rotation of second spur gear 736 .
- the second spur gear 736 is mechanically coupled to a lower gear portion 740 of the pin 716 .
- horizontal linear actuation of the rack gear 732 is translated to vertical linear actuation of the pin 716 .
- the actuator module 704 adjusts the tactile array 712 to represent contact between a user and a surface in the VR environment (e.g., responsive to commands from the control module 220 ).
- FIG. 8 Operation of an example contact device 800 is shown below in FIG. 8 .
- the contact device 800 is shown in a plurality of example configurations relative to a fingertip 804 of a user.
- the positions of the contact device 800 correspond to a contact point between a virtual fingertip of the user and a surface of an example object 808 in a VR environment.
- the virtual fingertip of the user e.g., as rendered and provided to the user via the display 212
- the virtual fingertip moves relative to the object 808 and contacts different points along the surface of the object 808 .
- the contact device 800 is adjusted as movement of the controller device 600 causes the virtual fingertip to make initial contact with the object 808 , contact different portions of the object 808 , discontinue contact with the object 808 , etc.
- actuator module 812 selectively extends and/or retracts pins 816 of tactile array 820 .
- the pins 816 are adjusted to a configuration such that a plane defined by the tactile array 820 is substantially perpendicular to a surface normal 816 of the object 808 at a contact point between the virtual fingertip and the object 808 .
- the pins 816 may be further adjusted based on a texture of the surface of the object 808 . For example, for an object 808 having a substantially flat and smooth surface, each of the pins 816 can be adjusted to a same height to provide a smooth surface. Conversely, for an object 808 having a rough surface, the pins 816 can be adjusted to different heights to provide a rough surface.
- the tactile array 820 may include a flexible or stretchable membrane arranged over the pins 816 .
- the controller devices 300 and 600 may implement additional features.
- the controller devices 300 and 600 may include a vibration mechanism (e.g., corresponding to the other outputs 244 of FIG. 2 ) configured to vibrate in response to user contact and/or lateral movement with a surface of an object in the VR environment.
- the vibration mechanism may be configured to vibrate as a function of movement speed of the controller device 300 or 600 relative to the surface of the object. In this manner, vibration may be indicative of the texture (e.g., roughness) of the surface.
- Other features may include, but are not limited to, a heater or other temperature output device to provide temperature feedback, inputs such as touchpads or force sensors incorporated within the respective contact devices 312 and 612 , etc.
- Some example VR systems may include two or more of the controller devices 300 or 600 , and/or include both the controller device 300 and the controller device 600 .
- An example method 900 for providing feedback to a user of a VR system begins at 904 .
- the method 900 adjusts a contact device of a controller (e.g., a handheld controller such as the controller device 300 or 600 ) to a default position.
- the default position may correspond to a fully lowered or retracted position with a neutral orientation relative to the controller.
- a planar surface of the contact device may be substantially parallel to a horizontal axis of the controller.
- the method 900 determines whether an appendage of a user (e.g., a fingertip) contacts a surface of an object in the VR environment. If true, the method 900 continues to 916 . If false, the method 900 continues to 908 .
- the method 900 adjusts the contact device based on the contact with the object. For example, the method 900 may adjust a position of the contact device based on a surface normal of the object at a contact point between the user and the object in the VR environment.
- the method 900 determines whether the contact point between the user and the object changed due to movement of the user. If true, the method 900 continues to 916 . If false, the method 900 continues to 924 .
- the method 900 maintains the position of the contact device.
- the method 900 determines whether the user has discontinued contact with the object in the VR environment. If true, the method 900 continues to 908 and returns the contact device to the default position. If false, the method 900 continues to 920 .
- Spatial and functional relationships between elements are described using various terms, including “connected,” “engaged,” “coupled,” “adjacent,” “next to,” “on top of,” “above,” “below,” and “disposed.” Unless explicitly described as being “direct,” when a relationship between first and second elements is described in the above disclosure, that relationship can be a direct relationship where no other intervening elements are present between the first and second elements, but can also be an indirect relationship where one or more intervening elements are present (either spatially or functionally) between the first and second elements.
- the phrase at least one of A, B, and C should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.”
- the direction of an arrow generally demonstrates the flow of information (such as data or instructions) that is of interest to the illustration.
- information such as data or instructions
- the arrow may point from element A to element B. This unidirectional arrow does not imply that no other information is transmitted from element B to element A.
- element B may send requests for, or receipt acknowledgements of, the information to element A.
- module or the term “controller” may be replaced with the term “circuit.”
- the term “module” may refer to, be part of, or include: an Application Specific Integrated Circuit (ASIC); a digital, analog, or mixed analog/digital discrete circuit; a digital, analog, or mixed analog/digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip.
- ASIC Application Specific Integrated Circuit
- FPGA field programmable gate array
- the module may include one or more interface circuits.
- the interface circuits may include wired or wireless interfaces that are connected to a local area network (LAN), the Internet, a wide area network (WAN), or combinations thereof.
- LAN local area network
- WAN wide area network
- the functionality of any given module of the present disclosure may be distributed among multiple modules that are connected via interface circuits. For example, multiple modules may allow load balancing.
- a server (also known as remote, or cloud) module may accomplish some functionality on behalf of a client module.
- code may include software, firmware, and/or microcode, and may refer to programs, routines, functions, classes, data structures, and/or objects.
- shared processor circuit encompasses a single processor circuit that executes some or all code from multiple modules.
- group processor circuit encompasses a processor circuit that, in combination with additional processor circuits, executes some or all code from one or more modules. References to multiple processor circuits encompass multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above.
- shared memory circuit encompasses a single memory circuit that stores some or all code from multiple modules.
- group memory circuit encompasses a memory circuit that, in combination with additional memories, stores some or all code from one or more modules.
- the term memory circuit is a subset of the term computer-readable medium.
- the term computer-readable medium does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); the term computer-readable medium may therefore be considered tangible and non-transitory.
- Non-limiting examples of a non-transitory, tangible computer-readable medium are nonvolatile memory circuits (such as a flash memory circuit, an erasable programmable read-only memory circuit, or a mask read-only memory circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).
- nonvolatile memory circuits such as a flash memory circuit, an erasable programmable read-only memory circuit, or a mask read-only memory circuit
- volatile memory circuits such as a static random access memory circuit or a dynamic random access memory circuit
- magnetic storage media such as an analog or digital magnetic tape or a hard disk drive
- optical storage media such as a CD, a DVD, or a Blu-ray Disc
- apparatus elements described as having particular attributes or performing particular operations are specifically configured to have those particular attributes and perform those particular operations.
- a description of an element to perform an action means that the element is configured to perform the action.
- the configuration of an element may include programming of the element, such as by encoding instructions on a non-transitory, tangible computer-readable medium associated with the element.
- the apparatuses and methods described in this application may be partially or fully implemented by a special purpose computer created by configuring a general purpose computer to execute one or more particular functions embodied in computer programs.
- the functional blocks, flowchart components, and other elements described above serve as software specifications, which can be translated into the computer programs by the routine work of a skilled technician or programmer.
- the computer programs include processor-executable instructions that are stored on at least one non-transitory, tangible computer-readable medium.
- the computer programs may also include or rely on stored data.
- the computer programs may encompass a basic input/output system (BIOS) that interacts with hardware of the special purpose computer, device drivers that interact with particular devices of the special purpose computer, one or more operating systems, user applications, background services, background applications, etc.
- BIOS basic input/output system
- the computer programs may include: (i) descriptive text to be parsed, such as HTML (hypertext markup language) or XML (extensible markup language), (ii) assembly code, (iii) object code generated from source code by a compiler, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc.
- source code may be written using syntax from languages including C, C++, C#, Objective C, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5, Ada, ASP (active server pages), PHP, Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, and Python®.
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- User Interface Of Digital Computer (AREA)
- Position Input By Displaying (AREA)
Abstract
Description
- The present disclosure relates to providing haptic feedback to a user of a controller device.
- The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent the work is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.
- Interactive computer systems implementing a virtual environment may include, but are not limited to, virtual reality (VR) systems, augmented reality systems, gaming systems, etc. For example only, VR systems provide a visual and audio representation of a three dimensional (3D) VR environment to a user. In some examples, the VR environment corresponds to a gaming environment. A VR system may provide the visual representation of the VR environment using a display, a headset incorporating a display, etc.
- The user may interact with the VR environment or other type of interactive computer system using a one or more controller devices. Controller devices include, but are not limited to, handheld (i.e., remote) controllers, gloves or other wearable devices, etc. configured to communicate with the VR system. For example, the controller devices may provide indications of respective positions (e.g., 6 degree-of-freedom, or 6-DOF, positions) of the controller devices to the VR system. The position indications may further indicate positions, orientations, movements, etc. of the respective hands holding the controller devices. In some examples, the VR system implements optical tracking to determine positions and movements of the controller devices. In this manner, the user is able to interact with (e.g., provide inputs to) the VR environment. The controller devices may also include other input mechanisms, such as buttons, touchpads, trackpads, etc. for receiving user inputs and causing further interaction with the VR environment.
- A controller device for a virtual environment includes a handle and a contact device having a substantially planar surface. A position of the contact device relative to the handle is adjustable. An actuator module is arranged to adjust the position of the contact device relative to the handle. A control module in communication with the virtual environment selectively controls the actuator module to adjust the position of the contact device in response to data received from the virtual environment. The data includes an indication of an interaction between a user and an object represented within the virtual environment.
- A method for operating a controller device for a virtual environment includes displaying an object within the virtual environment, generating data including an indication of interaction between a user and a surface of the object, providing the data to an actuator of the controller device, and using the actuator adjusting a position of a contact device relative to a handle of the controller device based on the data.
- Further areas of applicability of the present disclosure will become apparent from the detailed description, the claims and the drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.
-
FIG. 1 is an example virtual reality system according to the principles of the present disclosure. -
FIG. 2 is a simplified virtual reality system including a controller device shown in more detail according to the principles of the present disclosure. -
FIG. 3A is a side view of a first example controller device according to the principles of the present disclosure. -
FIG. 3B is a plan view of the first example controller device according to the principles of the present disclosure. -
FIGS. 4A and 4B are a first example contact device and actuator module according to the principles of the present disclosure. -
FIGS. 5A and 5B illustrate example operation of the first example controller device according to the principles of the present disclosure. -
FIG. 6A is a side view of a second example controller device according to the principles of the present disclosure. -
FIG. 6B is a plan view of the second example controller device according to the principles of the present disclosure. -
FIG. 7 is a second example contact device and actuator module according to the principles of the present disclosure. -
FIG. 8 illustrates example operation of the second example controller device according to the principles of the present disclosure. -
FIG. 9 is an example method for providing feedback to a user of a virtual reality system according to the principles of the present disclosure. - In the drawings, reference numbers may be reused to identify similar and/or identical elements.
- Some controller devices for virtual reality (VR) systems implement one or more mechanisms for providing feedback to the user. The feedback may be indicative of various features of a VR environment (e.g. proximity to objects, such as walls or other structures, the occurrence of an event within a certain range of the user, etc.) and/or responsive to user behavior within the VR environment (e.g., firing a virtual weapon, interacting with an object, etc.).
- Example feedback includes vibration, audio indicators (e.g., beeps), etc. The feedback provided to users via handheld or wearable controller devices may not accurately represent the interaction of the user with the VR environment. For example, feedback such as vibration may not accurately convey haptic interaction with objects in the VR environment to the user.
- Systems and methods according to the principles of the present disclosure provide haptic feedback to a user in response to interaction with objects, surfaces, etc. in the VR environment. For example, a controller device according to the principles of the present disclosure includes a contact device (e.g., a platform, pad, etc.) with a contact surface arranged to receive a fingertip of the user. The controller device is configured to adjust the contact surface of the contact device according to interaction between the user and the VR environment to provide haptic feedback (e.g., cutaneous and/or kinesthetic feedback). For example, in some implementations, the controller device adjusts an orientation of the contact surface in response to the user touching an object or surface in the VR environment. In other implementations, the controller device adjusts a texture of the contact surface. Although described with respect to VR systems, the principles of the present disclosure may also be included in other interactive computer systems implementing a virtual environment, including, but not limited to, augmented reality systems and gaming systems using a display such as a pc or laptop monitor, a gaming console using a television, etc. Accordingly, as used herein, the term “virtual reality system” may be used to indicate any interactive computer system implementing a virtual environment.
-
FIG. 1 shows anexample VR system 100. TheVR system 100 includes a host device (e.g., a computer, gaming console, etc.) 104 and one or more peripheral devices, such as acontroller device 108 according to the principles of the present disclosure and adisplay 112. In some examples, theVR system 100 may include a tracking system 116 (e.g., an optical tracking system) configured to track movements of a user and/or thecontroller device 108. TheVR system 100 may include other inputs 120 (e.g., a keyboard, mouse, wireless inputs, voice input mechanism such as a microphone, etc.) and other outputs 124 (e.g., speakers, secondary displays, wireless outputs, etc.). - The
host device 104 includes one ormore interfaces 128 for communicating with the peripheral devices. For example, theinterfaces 128 may include, but are not limited to, input/output (I/O) interfaces and adapters (e.g., universal serial bus, or USB), and wired or wireless communication interfaces (WiFi, Bluetooth, cellular, wired Ethernet, etc.) for communicating with anetwork 136. Theinterfaces 128 allow data to be transmitted between the peripheral devices and/or thenetwork 132 and acontrol module 136. - The
control module 136 controls processing of data related to operation of theVR system 100. For example, thecontrol module 136 may correspond to one or more processors configured to execute an operating system of thehost device 104 and one or more programs stored indisk storage 140,memory 144, a cloud computing system (e.g., via network 132), etc. to implement a VR environment. For example only, thedisk storage 140 may correspond to a hard disk drive (HDD), solid state drive (SSD), a removable media drive such as a CD or DVD ROM drive, flash memory, etc. Thememory 144 may correspond to system memory and may include, but it is not limited to, volatile and/or non-volatile semiconductor memory. -
FIG. 2 shows asimplified VR system 200 including only ahost device 204,controller device 208,display 212, andtracking system 216. Thecontroller device 208 includes acontrol module 220, acontact device 224,input sensors 228, andoutput actuators 232 configured to implement the systems and methods according to the principles of the present disclosure. Thecontroller device 208 includes communication interfaces (e.g., wired and/or wireless communication interfaces) 236 for communicating data between thecontroller device 208 and thehost device 204. Thecontroller device 208 may includeother inputs 240 andother outputs 244. For example, theother inputs 240 may include, but are not limited to, a microphone for receive voice inputs, buttons, a touchpad or touchscreen interface (e.g., for providing an indication of a relative position of a fingertip on the contact device 224), etc. Theother outputs 244 may include, but are not limited to, audio outputs, an integrated display, LEDs, other feedback mechanisms (e.g., vibration), etc. Thecontrol module 220 controls the operation of thecontroller device 208 based on data received from thehost device 204,contact device 224, andinputs 240, and provides data indicative of user interaction with thecontroller device 204 to thehost device 204 as described below in more detail. - The
host device 204 provides interaction data indicative of user interaction with the VR environment to the control module 22. For example, the user interacts with the VR environment (e.g., as presented to the user via the display 212) using thecontroller device 208. In some examples, thetracking system 216 tracks user movement by monitoring movement of thecontroller device 208 and provides tracking data indicative of the user movement to thehost device 204. Thehost device 204 generates and outputs the interaction data based on the tracking data received from the tracking system and, in some examples, further based on user input data received from the controller device 208 (e.g., user input data generated in response to user interaction withcontact device 224,inputs 240, etc.). - The
control module 220 controls thecontact device 224 based on the interaction data. For example, the interaction data may include characteristics of the interaction between the user and the VR environment. The characteristics may correspond to contact characteristics associated with contact between the user (e.g., a fingertip of the user in the VR environment) and an object in the VR environment, including, but not limited to, surface contour, shape, and texture of the object, as well as an amount of force exerted on and/or by the object. Accordingly, thecontrol module 220 controls theoutput actuators 232 to physically adjust thecontact device 224 so that contact between thecontact device 224 and an actual fingertip of the user reflects the contact between the fingertip of the user and the object in the VR environment. In some examples, input sensors 228 (which may include force or finger position sensors) provide additional feedback indicative of contact between the user and thecontact device 224. - Referring now to
FIGS. 3A and 3B and with continued reference toFIG. 2 , a firstexample controller device 300 according to the principles of the present disclosure is shown in a side view and a plan view, respectively. Thecontroller device 300 includes ahandle 304, anactuator module 308 arranged on a first end of thehandle 304, and acontact device 312 mounted on theactuator module 308. Thecontact device 308 is arranged to receive afingertip 316 of a user. Although thecontact device 308 as shown corresponds to a substantially circular platform, thecontact device 308 may have any suitable shape in other examples (e.g., rectangular, elliptical, etc.). In some examples, thecontroller device 300 may include one or more tracking elements 320 (e.g., retroreflective spheres) detectable by thetracking system 216 arranged to facilitate tracking of the movement of thecontroller device 300 in 3D space. Other examples may implement other suitable systems and methods for tracking movement of thecontroller device 300. For example only, components such as thecontrol module 220, theactuator module 308, etc. may be located within a housing of thehandle 304. - The
actuator module 308 selectively physically adjusts thecontact device 312 based on interaction data received from thehost device 204 as described above with respect toFIG. 2 . For example, theactuator module 308 may include servos or other suitable mechanisms in communication with respective actuator arms 324. Theactuator module 308 may be configured to independently actuate (e.g., extend and retract) the actuator arms 324 to adjust the position of (e.g., tilt to adjust yaw and pitch, raise, lower, etc.) thecontact device 312 relative to the handle 304 (and, therefore, relative to the fingertip 316). As the user moves thecontroller device 300 to cause contact between the fingertip of the user (i.e., a virtual fingertip) with an object in the VR environment, theactuator module 308 adjusts the position of thecontact device 312 accordingly. The visual representation of the fingertip of the user within the VR environment may also be adjusted to indicate the contact (e.g., by tilting or extruding the finger). - Referring now to
FIGS. 4A and 4B , anexample contact device 400 andactuator module 404 are shown. Thecontact device 400 includes substantially planar upper andlower platforms upper platform 408 may have a concaveupper surface 416. In some examples, theplatform 408 is formed from acetal or other plastic material. In some examples, a force sensor (e.g., a force transducer) 420 is arranged between the upper andlower platforms contact device 400 by a user. - The
actuator module 404 includesservo motors 424 configured to adjust thecontact device 400 via respective control arms 428 and actuator arms 432 (e.g., responsive to commands from the control module 220). As shown, theactuator module 404 includes threeservo motors 424, control arms 428, andactuator arms 432 to provide three degrees of freedom movement of thecontact device 400. For example only, the control arms 428 are connected to theactuator arms 432 via respectiverevolute joints 436 and theactuator arms 432 are connected to thecontact device 400 via ball-and-socketspherical joints 440. - For example only, the
force sensor 420 comprises a force sensing resistor (FSR) material and two internal electrodes for detecting compression between the upper andlower platforms upper platform 408 by the user. In some examples, theupper platform 408 is connected to thelower platform 412 via adjustable screws orother fasteners 444 at respective ends of radial arms 448 around an outer perimeter of theupper platform 408. For example, thearms 444 may be formed in the material of theupper platform 408 to allow downward movement of an inner portion of theupper platform 408 in response to force applied by the user. - When no force is applied to the
upper platform 408, electrodes of theforce sensor 420 do not contact each other and no voltage is generated. Conversely, a minimal amount of force applied to theupper platform 408 causes contact between the electrodes of theforce sensor 420, and a voltage is generated accordingly to indicate an amount of force applied to thecontact device 400 by the user. In some examples, thearms 444 may be configured to bias theupper platform 408 downward against theforce sensor 420 even without contact from a user. Accordingly, a nominal voltage may be generated by theforce sensor 420 with no user contact, while the voltage may increase proportionately to an amount of force applied to theupper platform 408 by the user. The voltage is provided as feedback indicative of the amount of force applied by the user (e.g., to the control module 220). - Operation of an
example contact device 500 is described below with respect toFIGS. 5A and 5B . InFIG. 5A , thecontact device 500 is shown in a plurality of example positions relative to afingertip 504 of a user. As shown, the positions of thecontact device 500 correspond to a contact point between a virtual fingertip of the user and a surface of anexample object 508 in a VR environment. For example, the virtual fingertip of the user (e.g., as rendered and provided to the user via the display 212) moves within the VR environment based on movement of thecontroller device 300. Accordingly, as the user moves thecontroller device 300, the virtual fingertip moves relative to theobject 508 and contacts different points along the surface of theobject 508. Thecontact device 500 is adjusted (e.g., tilted) as movement of thecontroller device 300 causes the virtual fingertip to make initial contact with theobject 508, contact different portions of theobject 508, discontinue contact with theobject 508, etc. For example, as the virtual fingertip “slides” along the surface of theobject 508, thecontact device 500 is tilted accordingly (e.g. relative to thecontroller device 300, a mounting portion such asactuator module 512, etc.). For example only, thecontact device 500 is adjusted such that thecontact device 500 is substantially perpendicular to a surface normal 516 of theobject 508 at a contact point between the virtual fingertip and theobject 508. - In
FIG. 5B , thecontact device 500 is shown in a plurality of positions (e.g., height positions) as a virtual fingertip of a user contacts asurface 520 in the VR environment. At 524, the virtual fingertip is not in contact with thesurface 520 and therefore thecontact device 500 is shown in a fully retracted position. At 528, the virtual fingertip is in contact with thesurface 520. - Continued downward movement of the
controller device 300 may cause the virtual fingertip to penetrate thesurface 520. Thecontact device 500 according to the principles of the present disclosure may be extended in response to movement that would otherwise cause the virtual fingertip to penetrate thesurface 520 within the VR environment. For example, at 532, thecontact device 500 is extended to apply upward force against thefingertip 504 of the user. In other words, as thecontroller device 300 is moved downward, thefingertip 504 is nonetheless maintained in a same position due to the opposing, upward movement of thecontact device 500. In this manner, thecontact device 500 indicates to the user that the movement of thecontroller device 300 is causing the virtual fingertip to attempt to penetrate thesurface 520. - If user behavior continues to cause the virtual fingertip to move downward toward the
surface 520, thecontact device 500 is extended still further as shown at 536. The position of thecontact device 500 as shown at 536 may correspond to a fully extended position. Accordingly, further downward movement of thecontroller device 300 allows corresponding movement of thefingertip 504, resulting in penetration of thesurface 520 as shown at 540. In some examples, a position of the virtual fingertip (and a virtual hand, which corresponds to a position of the controller device 300) as rendered in the VR environment may be decoupled from further movement of thecontroller device 300 to compensate for penetration of thesurface 520. For example, rather than allowing the virtual fingertip to penetrate thesurface 520 in response to continued movement of thecontroller device 300, thecontact device 500 may be maintained in a partially extended position (e.g., 50% or 75% of a fully extended position) while maintaining presentation of the virtual fingertip on thesurface 520. Accordingly, if penetration of thesurface 520 by the user is inadvertent, the user is nonetheless able to interact with thesurface 520. - Referring now to
FIGS. 6A and 6B , a secondexample controller device 600 according to the principles of the present disclosure is shown in a side view and a plan view, respectively. Thecontroller device 600 includes ahandle 604, anactuator module 608 arranged on a first end of thehandle 604, and acontact device 612 mounted on theactuator module 308. Thecontact device 608 is arranged to receive afingertip 616 of a user. Although thecontact device 608 as shown corresponds to a substantially circular platform, thecontact device 608 may have any suitable shape in other examples (e.g., rectangular, elliptical, etc.). In some examples, thecontroller device 600 may include one or more tracking elements 620 (e.g., retroreflective spheres) detectable by thetracking system 216 arranged to facilitate tracking of the movement of thecontroller device 600. For example only, components such as thecontrol module 220, theactuator module 608, etc. may be located within a housing of thehandle 604. - The
actuator module 608 selectively adjusts thecontact device 612 based on interaction data received from thehost device 204 as described above with respect toFIG. 2 . In this example, thecontact device 612 includes a tactile array 624 (e.g., as shown, a 4×4 array) ofadjustable pins 628. Theactuator module 608 includes servos and/or other suitable mechanisms configured to independently actuate respective ones of thepins 628. For example, theactuator module 608 independently actuates (e.g., extends and retracts) thepins 628 to adjust the position of (e.g., tilt, raise, lower, etc.) thetactile array 624 relative to the handle 604 (and, therefore, relative to the fingertip 616). As the user moves thecontroller device 600 to cause contact between the fingertip of the user (i.e., a virtual fingertip) with an object in the VR environment, theactuator module 608 adjusts respective positions of thepins 628 of thetactile array 624 accordingly. - Referring now to
FIG. 7 , an example contact device 700 andactuator module 704 are shown. The contact device 700 includes a substantially planarupper platform 708 including a 4 x 4tactile array 712 of sixteenpins 716. Although thetactile array 712 has a 4×4 configuration, other examples may use a different configuration (e.g., 3×4, 5×5, 4×6, etc.). Theactuator module 704 is configured to independently actuate (e.g., extend and retract) each of thepins 716 based on interaction with an object in the VR environment. - For example, the
actuator module 704 includes a plurality of linear actuator systems 720 in communication with respective ones of thepins 716, although only a single linear actuator system 720 is shown for simplicity. For example only, the linear actuator system 720 has a rack and pinion configuration. Each of the linear actuator systems 720 includes aservo motor 724 configured to rotate a first spur orpinion gear 728. Rotation of thefirst spur gear 728 causes linear actuation ofrack gear 732, which in turn causes rotation ofsecond spur gear 736. Thesecond spur gear 736 is mechanically coupled to alower gear portion 740 of thepin 716. Accordingly, horizontal linear actuation of therack gear 732 is translated to vertical linear actuation of thepin 716. In this manner, theactuator module 704 adjusts thetactile array 712 to represent contact between a user and a surface in the VR environment (e.g., responsive to commands from the control module 220). - Operation of an
example contact device 800 is shown below inFIG. 8 . Thecontact device 800 is shown in a plurality of example configurations relative to afingertip 804 of a user. As shown, the positions of thecontact device 800 correspond to a contact point between a virtual fingertip of the user and a surface of anexample object 808 in a VR environment. For example, the virtual fingertip of the user (e.g., as rendered and provided to the user via the display 212) moves within the VR environment based on movement of thecontroller device 600. Accordingly, as the user moves thecontroller device 600, the virtual fingertip moves relative to theobject 808 and contacts different points along the surface of theobject 808. Thecontact device 800 is adjusted as movement of thecontroller device 600 causes the virtual fingertip to make initial contact with theobject 808, contact different portions of theobject 808, discontinue contact with theobject 808, etc. - For example, as the virtual fingertip “slides” along the surface of the
object 808,actuator module 812 selectively extends and/or retractspins 816 oftactile array 820. For example only, thepins 816 are adjusted to a configuration such that a plane defined by thetactile array 820 is substantially perpendicular to a surface normal 816 of theobject 808 at a contact point between the virtual fingertip and theobject 808. In some examples, thepins 816 may be further adjusted based on a texture of the surface of theobject 808. For example, for anobject 808 having a substantially flat and smooth surface, each of thepins 816 can be adjusted to a same height to provide a smooth surface. Conversely, for anobject 808 having a rough surface, thepins 816 can be adjusted to different heights to provide a rough surface. In some examples, thetactile array 820 may include a flexible or stretchable membrane arranged over thepins 816. - The
controller devices controller devices other outputs 244 ofFIG. 2 ) configured to vibrate in response to user contact and/or lateral movement with a surface of an object in the VR environment. For example, the vibration mechanism may be configured to vibrate as a function of movement speed of thecontroller device respective contact devices controller devices controller device 300 and thecontroller device 600. - An
example method 900 for providing feedback to a user of a VR system according to the principles of the present disclosure begins at 904. At 908, themethod 900 adjusts a contact device of a controller (e.g., a handheld controller such as thecontroller device 300 or 600) to a default position. For example, the default position may correspond to a fully lowered or retracted position with a neutral orientation relative to the controller. For example only, in the default position, a planar surface of the contact device may be substantially parallel to a horizontal axis of the controller. - At 912, the
method 900 determines whether an appendage of a user (e.g., a fingertip) contacts a surface of an object in the VR environment. If true, themethod 900 continues to 916. If false, themethod 900 continues to 908. At 916, themethod 900 adjusts the contact device based on the contact with the object. For example, themethod 900 may adjust a position of the contact device based on a surface normal of the object at a contact point between the user and the object in the VR environment. At 920, themethod 900 determines whether the contact point between the user and the object changed due to movement of the user. If true, themethod 900 continues to 916. If false, themethod 900 continues to 924. At 924, themethod 900 maintains the position of the contact device. - At 928, the
method 900 determines whether the user has discontinued contact with the object in the VR environment. If true, themethod 900 continues to 908 and returns the contact device to the default position. If false, themethod 900 continues to 920. - The foregoing description is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses. The broad teachings of the disclosure can be implemented in a variety of forms. Therefore, while this disclosure includes particular examples, the true scope of the disclosure should not be so limited since other modifications will become apparent upon a study of the drawings, the specification, and the following claims. It should be understood that one or more steps within a method may be executed in different order (or concurrently) without altering the principles of the present disclosure. Further, although each of the embodiments is described above as having certain features, any one or more of those features described with respect to any embodiment of the disclosure can be implemented in and/or combined with features of any of the other embodiments, even if that combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and permutations of one or more embodiments with one another remain within the scope of this disclosure.
- Spatial and functional relationships between elements (for example, between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including “connected,” “engaged,” “coupled,” “adjacent,” “next to,” “on top of,” “above,” “below,” and “disposed.” Unless explicitly described as being “direct,” when a relationship between first and second elements is described in the above disclosure, that relationship can be a direct relationship where no other intervening elements are present between the first and second elements, but can also be an indirect relationship where one or more intervening elements are present (either spatially or functionally) between the first and second elements. As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.”
- In the figures, the direction of an arrow, as indicated by the arrowhead, generally demonstrates the flow of information (such as data or instructions) that is of interest to the illustration. For example, when element A and element B exchange a variety of information but information transmitted from element A to element B is relevant to the illustration, the arrow may point from element A to element B. This unidirectional arrow does not imply that no other information is transmitted from element B to element A. Further, for information sent from element A to element B, element B may send requests for, or receipt acknowledgements of, the information to element A.
- In this application, including the definitions below, the term “module” or the term “controller” may be replaced with the term “circuit.” The term “module” may refer to, be part of, or include: an Application Specific Integrated Circuit (ASIC); a digital, analog, or mixed analog/digital discrete circuit; a digital, analog, or mixed analog/digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip.
- The module may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces that are connected to a local area network (LAN), the Internet, a wide area network (WAN), or combinations thereof. The functionality of any given module of the present disclosure may be distributed among multiple modules that are connected via interface circuits. For example, multiple modules may allow load balancing. In a further example, a server (also known as remote, or cloud) module may accomplish some functionality on behalf of a client module.
- The term code, as used above, may include software, firmware, and/or microcode, and may refer to programs, routines, functions, classes, data structures, and/or objects. The term shared processor circuit encompasses a single processor circuit that executes some or all code from multiple modules. The term group processor circuit encompasses a processor circuit that, in combination with additional processor circuits, executes some or all code from one or more modules. References to multiple processor circuits encompass multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term shared memory circuit encompasses a single memory circuit that stores some or all code from multiple modules. The term group memory circuit encompasses a memory circuit that, in combination with additional memories, stores some or all code from one or more modules.
- The term memory circuit is a subset of the term computer-readable medium. The term computer-readable medium, as used herein, does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); the term computer-readable medium may therefore be considered tangible and non-transitory. Non-limiting examples of a non-transitory, tangible computer-readable medium are nonvolatile memory circuits (such as a flash memory circuit, an erasable programmable read-only memory circuit, or a mask read-only memory circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).
- In this application, apparatus elements described as having particular attributes or performing particular operations are specifically configured to have those particular attributes and perform those particular operations. Specifically, a description of an element to perform an action means that the element is configured to perform the action. The configuration of an element may include programming of the element, such as by encoding instructions on a non-transitory, tangible computer-readable medium associated with the element.
- The apparatuses and methods described in this application may be partially or fully implemented by a special purpose computer created by configuring a general purpose computer to execute one or more particular functions embodied in computer programs. The functional blocks, flowchart components, and other elements described above serve as software specifications, which can be translated into the computer programs by the routine work of a skilled technician or programmer.
- The computer programs include processor-executable instructions that are stored on at least one non-transitory, tangible computer-readable medium. The computer programs may also include or rely on stored data. The computer programs may encompass a basic input/output system (BIOS) that interacts with hardware of the special purpose computer, device drivers that interact with particular devices of the special purpose computer, one or more operating systems, user applications, background services, background applications, etc.
- The computer programs may include: (i) descriptive text to be parsed, such as HTML (hypertext markup language) or XML (extensible markup language), (ii) assembly code, (iii) object code generated from source code by a compiler, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. As examples only, source code may be written using syntax from languages including C, C++, C#, Objective C, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5, Ada, ASP (active server pages), PHP, Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, and Python®.
- None of the elements recited in the claims are intended to be a means-plus-function element within the meaning of 35 U.S.C. §112(f) unless an element is expressly recited using the phrase “means for,” or in the case of a method claim using the phrases “operation for” or “step for.”
Claims (22)
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/255,866 US9916003B1 (en) | 2016-09-02 | 2016-09-02 | 3D haptics for interactive computer systems |
PCT/US2017/048099 WO2018044634A1 (en) | 2016-09-02 | 2017-08-23 | 3d haptics for interactive computer systems |
CN201780053887.2A CN109661640B (en) | 2016-09-02 | 2017-08-23 | 3D haptic sensation for interactive computer systems |
EP17761981.4A EP3507678B1 (en) | 2016-09-02 | 2017-08-23 | 3d haptics for interactive computer systems |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/255,866 US9916003B1 (en) | 2016-09-02 | 2016-09-02 | 3D haptics for interactive computer systems |
Publications (2)
Publication Number | Publication Date |
---|---|
US20180067543A1 true US20180067543A1 (en) | 2018-03-08 |
US9916003B1 US9916003B1 (en) | 2018-03-13 |
Family
ID=59791155
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/255,866 Active US9916003B1 (en) | 2016-09-02 | 2016-09-02 | 3D haptics for interactive computer systems |
Country Status (4)
Country | Link |
---|---|
US (1) | US9916003B1 (en) |
EP (1) | EP3507678B1 (en) |
CN (1) | CN109661640B (en) |
WO (1) | WO2018044634A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190041988A1 (en) * | 2018-05-02 | 2019-02-07 | Intel IP Corporation | Deformable objects for haptic feedback |
US20220409996A1 (en) * | 2021-06-24 | 2022-12-29 | Microsoft Technology Licensing, Llc | Mobile Haptic Robots |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10572016B2 (en) * | 2018-03-06 | 2020-02-25 | Microsoft Technology Licensing, Llc | Spatialized haptic device force feedback |
Family Cites Families (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5803738A (en) | 1994-06-24 | 1998-09-08 | Cgsd Corporation | Apparatus for robotic force simulation |
US6184868B1 (en) | 1998-09-17 | 2001-02-06 | Immersion Corp. | Haptic feedback control devices |
US7098888B2 (en) | 2000-04-28 | 2006-08-29 | Texas Tech University System | Development of stereoscopic-haptic virtual environments |
US20060209019A1 (en) | 2004-06-01 | 2006-09-21 | Energid Technologies Corporation | Magnetic haptic feedback systems and methods for virtual reality environments |
US20050285854A1 (en) | 2004-06-29 | 2005-12-29 | Ge Medical Systems Information Technologies, Inc. | 3D display system and method |
WO2006081198A2 (en) | 2005-01-25 | 2006-08-03 | The Board Of Trustees Of The University Of Illinois | Compact haptic and augmented virtual reality system |
US11325029B2 (en) | 2007-09-14 | 2022-05-10 | National Institute Of Advanced Industrial Science And Technology | Virtual reality environment generating apparatus and controller apparatus |
JP5401838B2 (en) * | 2007-11-28 | 2014-01-29 | ソニー株式会社 | Tactile sheet member, input device, and electronic device |
US8154529B2 (en) * | 2009-05-14 | 2012-04-10 | Atmel Corporation | Two-dimensional touch sensors |
US8542105B2 (en) * | 2009-11-24 | 2013-09-24 | Immersion Corporation | Handheld computer interface with haptic feedback |
BR112013011300A2 (en) * | 2010-11-09 | 2019-09-24 | Koninl Philips Electronics Nv | user interface, method of providing the user with tactile feedback that touches an interaction surface (s) with a set of drivers and apparatus comprising a user interface |
US9298264B2 (en) * | 2011-05-16 | 2016-03-29 | Massachusetts Institute Of Technology | Methods and apparatus for actuated 3D surface with gestural interactivity |
US9142105B1 (en) | 2012-06-01 | 2015-09-22 | Jonathan M. Crofford | Haptic device capable of managing distributed force |
US9563266B2 (en) | 2012-09-27 | 2017-02-07 | Immersivetouch, Inc. | Haptic augmented and virtual reality system for simulation of surgical procedures |
US9715300B2 (en) | 2013-03-04 | 2017-07-25 | Microsoft Technology Licensing, Llc | Touch screen interaction using dynamic haptic feedback |
CN105144035B (en) * | 2013-04-26 | 2019-03-22 | 意美森公司 | Utilize the simulation Tangible User Interfaces interaction of haptic unit array and gesture |
CN105094310B (en) | 2014-05-09 | 2019-08-16 | 梦工厂动画公司 | Carsick method and system is reduced in virtual reality driving system |
US20170124767A1 (en) * | 2015-11-02 | 2017-05-04 | Scott Everitt Foust | Dynamic Surfaces for Virtual Reality Applications |
US10509468B2 (en) * | 2016-01-27 | 2019-12-17 | Tactai, Inc. | Providing fingertip tactile feedback from virtual objects |
-
2016
- 2016-09-02 US US15/255,866 patent/US9916003B1/en active Active
-
2017
- 2017-08-23 CN CN201780053887.2A patent/CN109661640B/en active Active
- 2017-08-23 EP EP17761981.4A patent/EP3507678B1/en active Active
- 2017-08-23 WO PCT/US2017/048099 patent/WO2018044634A1/en unknown
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190041988A1 (en) * | 2018-05-02 | 2019-02-07 | Intel IP Corporation | Deformable objects for haptic feedback |
US10845879B2 (en) * | 2018-05-02 | 2020-11-24 | Intel IP Corporation | Deformable objects for haptic feedback |
US20220409996A1 (en) * | 2021-06-24 | 2022-12-29 | Microsoft Technology Licensing, Llc | Mobile Haptic Robots |
Also Published As
Publication number | Publication date |
---|---|
EP3507678B1 (en) | 2020-09-09 |
EP3507678A1 (en) | 2019-07-10 |
CN109661640A (en) | 2019-04-19 |
US9916003B1 (en) | 2018-03-13 |
WO2018044634A1 (en) | 2018-03-08 |
CN109661640B (en) | 2022-05-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP3105660B1 (en) | Low-profile pointing stick | |
US10514780B2 (en) | Input device | |
US10877559B2 (en) | System to provide tactile feedback during non-contact interaction | |
US10234944B2 (en) | Force feedback system including multi-tasking graphical host environment | |
US9971405B2 (en) | Touch sensitive input | |
EP3507678B1 (en) | 3d haptics for interactive computer systems | |
KR102060338B1 (en) | Apparatus and method for providing three dimensions tactile | |
EP3105662B1 (en) | Low-profile pointing stick | |
US9189088B2 (en) | Electronic device and touch operating method thereof | |
KR20170083377A (en) | Electronic Apparatus and Controlling Method thereof | |
US20150261330A1 (en) | Method of using finger surface area change on touch-screen devices - simulating pressure | |
US20170351393A1 (en) | Auto targeting assistance for input devices | |
US10016894B2 (en) | Method and apparatus for controlling a robot | |
CN111566587B (en) | Changing control rod moving space | |
KR101879531B1 (en) | Touch input appratus displaying interface on a touchpad performing various functions | |
US10589428B1 (en) | Automatic distribution of control functions between multiple knob controllers based on proximity and relative positioning of the knobs | |
CN117222966A (en) | touch sensitive input device | |
US20240103643A1 (en) | Input device | |
KR200313326Y1 (en) | Cursor controlling system | |
KR100496503B1 (en) | Cursor controlling system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SINCLAIR, MICHAEL JACK;BENKO, HRVOJE;HOLZ, CHRISTIAN;SIGNING DATES FROM 20160824 TO 20160826;REEL/FRAME:039624/0213 |
|
AS | Assignment |
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:OFEK, EYAL;REEL/FRAME:039669/0744 Effective date: 20160907 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |