EP4457587A1 - Device and method for providing an application user interface with haptic feedback - Google Patents

Device and method for providing an application user interface with haptic feedback

Info

Publication number
EP4457587A1
EP4457587A1 EP21854756.0A EP21854756A EP4457587A1 EP 4457587 A1 EP4457587 A1 EP 4457587A1 EP 21854756 A EP21854756 A EP 21854756A EP 4457587 A1 EP4457587 A1 EP 4457587A1
Authority
EP
European Patent Office
Prior art keywords
haptic
providing
haptic feedback
user interface
application user
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP21854756.0A
Other languages
German (de)
French (fr)
Inventor
Pedro Miguel DE SOUSA BERNARDO
Vera Patricia CASTRO VILAS BOAS
Nélson Bruno MARTINS MARQUES COSTA
Manuel Sebastião MARQUES COUTINHO
Adriano Dídimo MACHADO CARVALHO
Rosane SCHONBLUM SAMPAIO
Marina MACHADO
Lisandra Maria ARAÚJO TEIXEIRA
Emanuel TEIXEIRA SILVA
José Pedro CORREIA DIAS
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Universidade do Minho
Bosch Car Multimedia Portugal SA
Original Assignee
Universidade do Minho
Bosch Car Multimedia Portugal SA
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Universidade do Minho, Bosch Car Multimedia Portugal SA filed Critical Universidade do Minho
Publication of EP4457587A1 publication Critical patent/EP4457587A1/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/01—Indexing scheme relating to G06F3/01
    • G06F2203/014—Force feedback applied to GUI
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/038—Indexing scheme relating to G06F3/038
    • G06F2203/0381—Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer

Definitions

  • the present disclosure relates to a device and method for providing an application user interface with haptic feedback for a passenger in an automotive setting.
  • CN213092557U discloses a sense of touch that can speak turns over voiced book, including the book body, the internal page or leaf in a plurality of books that is provided with of book, all printed the characters pattern on the page or leaf in every book, the book body is including left book body and right book body, left side book body and right book body connect and be the books form, and left book body can be followed left book body and right book body junction and opened and closed, right side book body is equipped with sound generating mechanism by right side edge, sound generating mechanism includes the sound generating mechanism shell and sets up loudspeaker and the PCB board in the sound generating mechanism shell, loudspeaker and PCB board electric connection, be provided with volume on the sound generating mechanism shell and keep off the position switch, be provided with book page or leaf switch in this volume below that keeps off the position switch, page or leaf switch and volume keep off the position switch and all are connected with the PCB board in the book.
  • the utility model discloses simple structure, space utilization is high, and is small, conveniently carries, and page or
  • KR20160010843A discloses a method for playing an audio book provided with a vibration function, comprising the steps of: loading sound source data to play the audio book and obtaining total play time of the sound source data loaded; playing the sound source data; loading one or more from vibration data and illustration data corresponding to the sound source data; obtaining one or more from the vibration information and the illustration information corresponding to the play time of the sound source data from one or more from the vibration data and the illustration data; and outputting vibration or illustration based on one or more from the vibration data or the illustration data obtained.
  • US2016240102A1 discloses a system for enhanced learning that combines tactile surfaces and audio recordings, allowing the user to explore a 3-dimensional object through touch while also hearing pre-recorded audio explaining regions or features of interest.
  • One of a collection of 3-dimensional tactile modules is used with the system at a given time.
  • the system automatically recognizes the inserted module, locates the set of audio recordings and region maps associated with that tactile module, and then plays module-specific audio recordings.
  • Document EP2972687A1 discloses a method of receiving user input includes operating a software program in a first operating mode and modifying a graphical interface of the software program in response to receiving gesture. The method also includes receiving a haptic input and operating the software program in a second operating mode in response to the haptic input. In the second operating mode, the gesture produces a different modification to the graphical interface than in the first operating mode.
  • the present disclosure relates to a device and method for providing an application user interface with haptic feedback for a passenger in an automotive setting.
  • One of the main ways in which children explore the world around them is through touch. When human physicality is used, especially the sense of touch, the brain receives a higher amount of stimulation. Thus, by physically interacting with "things", children acquire learning experiences that provide important contributions for their development, with a positive impact on their comprehension of the world around them (Shibasaki et al., 2020).
  • haptic feedback modalities By introducing haptic feedback modalities into drawing activities, richer, more immersive, and more entertaining experiences can be created and presented to children, which can have positive impacts on their development. All these activities, among others, that are enhanced by the presence of haptic feedback sensations, can be either partially or fully implemented as applications (apps) in modern entertainment systems, such as tablets and computers. With current display and surface technologies, devices can be made so that their touchscreen surfaces have the ability to provide users with different tactile feedback sensations beyond those that are normally felt when touching the surface, such as vibrotactile (KHOSHKAVA et al., 2019) and frictional (Olley et al., 2020) stimuli.
  • An aspect of the present disclosure relates to making use of a smart surface, composed of an insulative layer, fitted atop a transparent and conductive electrode sheet layer, which is in turn fitted atop a touch sensitive layer.
  • This touch-sensitive layer is fitted atop a layer of LED lights, which are in turn fitted atop a layer of vibrotactile actuators.
  • the layer of LED lights is optional, and when not presented, the touch-sensitive layer is fitted atop the layer of vibrotactile actuators.
  • This smart surface can be employed either as: a) the interactive surface interface of a display device to which it is physically connect (e.g., the smart surface is part of the device's display); or b) as the interactive surface interface of a display device to which it is not physically connected (e.g., touch surface interface communicating wirelessly with a display device in close proximity).
  • the capacitive-based touch sensitive layer can be made of a variety of materials, depending on the intended use case and aesthetic appeal (e.g., transparent glass - display device; translucent plastic - interactive surface interface for an external device with LED lights underneath it). This smart surface can generate both frictional and vibrotactile haptic sensations.
  • Frictional haptic sensations are generated thanks to the insulative and conductive layers, which allow for friction between the smart surface and the user's skin to be modulated through electrostatic actuation, while the skin is in motion.
  • Vibrotactile haptic sensations are generated thanks to the vibrotactile actuators, which can generate vibrations strong enough to be felt through the user's skin, either stationary or in motion, when in direct contact with the smart surface.
  • haptic feedback sensations caused by the external device's software, can be generated on it, as long as the haptic actuation technology that is present on the smart surface is both capable of interpreting the software's information and of generating such sensations.
  • this smart surface is to be fitted on at least one region inside the vehicle: the display surface of an entertainment display on the back of vehicle, where it will serve as said device's interactive surface interface.
  • This device can be attacked to an individual adjustable gooseneck holder, which can be extended and manipulated so that it is placed at a comfortable location for the user to interact with.
  • Other entertainment displays inside the vehicle can also be fitted with said smart surface, such as other entertainment display devices available for rear passengers, but the use of the disclosure should be preferably limited according to the above-mentioned restrictions regarding child passengers.
  • a device for providing an application user interface with haptic feedback for a passenger in an automotive setting comprising: a haptic surface comprising actuators for exhibiting tactile textures, comprising: a touch sensitive layer, a frictional haptic layer comprising an insulative sheet for user touch and a transparent conductive electrode film for providing frictional haptic sensations, and a vibrotactile haptic layer for providing vibrotactile haptic sensations; a display for displaying images of objects of said application user surface; an electronic data processor configured to: drive the haptic layers to exhibit a tactile texture when the user touches the image of an object in said display; to provide haptic feedback as the user touches the device.
  • the touch sensitive layer is a capacitive-based touch sensitive layer.
  • the electronic data processor is further configured to drive the haptic layers to exhibit a tactile texture whose roughness is defined dependent of a colour of the touched object in said display.
  • the electronic data processor is further configured to drive the haptic layers to exhibit a tactile texture whose roughness is defined dependent of the touched object being within or outside a predetermined boundary in said display.
  • the electronic data processor is further configured to drive the haptic layers to exhibit a tactile texture whose roughness is defined dependent of the touched object being along or outside a predetermined path in said display.
  • the electronic data processor is further configured to play an audio file or an audio streaming when the object is touched.
  • the touch sensitive layer is made of translucent plastic comprising LEDs underneath it.
  • the automotive setting is an autonomous vehicle and the display is an entertainment display arranged to be mounted on the back of the vehicle.
  • the display is an entertainment display arranged to be mounted on the back of the vehicle.
  • In an embodiment comprises an adjustable gooseneck holder or a headrest tablet holder for supporting the device.
  • Figure 1 illustrates a schematic illustration of a child user on back seat using app on haptic display device according to an embodiment.
  • Figure 2 illustrates a schematic illustration of an application home screen (different waveform indicate different texture sensations) according to an embodiment.
  • Figure 3 illustrates a schematic illustration of a drawing with different colours (different waveform indicate different texture sensations) according to an embodiment.
  • Figure 4 illustrates a schematic illustration of a drawing inside and outside the lines (different waveform indicate different texture sensations) according to an embodiment.
  • Figure 5 illustrates a schematic illustration of a copying a item to learn how to write it (different waveform indicate different texture sensations) according to an embodiment.
  • Figure 6 shows a schematic illustration of an animal picture dictionary, exemplary page (different waveform indicate different texture sensations) according to an embodiment.
  • Figure 7 shows a schematic illustration of a haptic sensations which then triggers from and up to certain "trigger" words (different waveform indicate different texture sensations) according to an embodiment.
  • the present disclosure relates to a device for providing an application user interface with haptic feedback for a passenger in an automotive setting, comprising: a haptic surface comprising actuators for exhibiting tactile textures, comprising: a touch sensitive layer, a frictional haptic layer comprising an insulative sheet for user touch and a transparent conductive electrode film for providing frictional haptic sensations, and a vibrotactile haptic layer for providing vibrotactile haptic sensations; a display for displaying images of objects of said application user surface; an electronic data processor configured to: drive the haptic layers to exhibit a tactile texture when the user touches the image of an object in said display; to provide haptic feedback as the user touches the device.
  • Figure 1 shows a schematic representation of an embodiment of a child user on back seat using app on haptic display device 1 according to an embodiment, for providing a user interface 2.
  • the haptic feedback children edutainment app can be installed and accessed through several different devices, such as the vehicle's infotainment system or the user's personal smartphone, as long as the software permits it.
  • the full benefits of the haptic feedback sensations described below can only be felt when the interaction between user and app is done through a device or surface which possesses actuators capable of generating the intended sensations. Therefore, the interaction between user and app is more fruitful when done through a smart surface as , especially when said interaction occurs inside the vehicle, during the user's commute.
  • haptic feedback sensations described in this section make use of frictional and/vibrotactile stimuli, which can be generated through electrostatic and vibrotactile actuation, respectively.
  • the main innovation of this app is the employment of haptic feedback sensations during the interactions between it and the user, there are no impediments regarding installing and using this app on devices without actuators that can produce the stimuli types from which this app benefits, as long as users keep in mind that the full intended experience cannot be achieved in said devices.
  • a free drawing mode in which users are presented with an empty, unrestricted canvas, where they can freely draw and paint with their fingers, using a variety of colours, with each colour giving their own distinct haptic feedback sensation (e.g., haptic feedback ranging from rougher to smoother, depending on the colour's RGB values) (Figure 3).
  • the brush tool can be disengaged so that users can freely explore their creations through touch controls and by sliding their fingers across the display surface.
  • the image can be saved in the device's memory to be loaded later or exported through a USB or wireless connection to another device (if available), as not to restrict the user's creation to only being available on the vehicle's displays.
  • This mode is mainly aimed at users around or above 4 years old;
  • FIG. 4 A figure drawing and painting mode, in which the outline of different items (e.g., animals, objects, fruits) will be displayed, with the user being tasked with painting inside the outlines.
  • two extra modes are available, one, “Original”, in which the correct colours should be used for the corresponding portion, and another, “Freestyle”, in which the user is free to use whatever colour they wish.
  • each colour will give a corresponding haptic feedback sensation.
  • a rougher haptic sensation is felt to indicate to the user that they are painting out of bounds ( Figure 4).
  • the brush tool can be disengaged so that users can freely explore their creations through touching or sliding their fingers.
  • the image can be saved in the device's memory to be loaded later, or exported through a USB or wireless connection to another device, to not restrict the user's creation to only the in-vehicle display.
  • This mode is mainly aimed at users around or above 4 years old;
  • flashcards containing the visualization of different elements (e.g., roman numerals, Arabic numerals, Latin alphabet letters, Hiragana, Kanjis) are displayed on either the right or the left side of the screen, one at a time, alongside an audio file containing the correct vocalization of said element, which is played when the item is loaded in.
  • the user's task in this mode, is to draw the visualization of the element present on the screen on the other portion of the screen not displaying the element (which side is the visualization and the drawing portion of the screen can be adjusted in settings by the parent or supervising adult, depending on if the user is left-handed, righthanded, or ambidextrous).
  • This mode is mainly aimed at users above 4 years old, with different settings available depending on the user's age (e.g., 4-year-olds - all Latin alphabet letters are displayed in their uppercase form, by themselves, and Arabic numbers from 1 to 10 are presented; 6-year-olds - Latin alphabet letters can be displaying in uppercase and lowercase form, alongside other letters to form words, and Arabic numbers from 1 to 100 are presented).
  • 4-year-olds - all Latin alphabet letters are displayed in their uppercase form, by themselves, and Arabic numbers from 1 to 10 are presented; 6-year-olds - Latin alphabet letters can be displaying in uppercase and lowercase form, alongside other letters to form words, and Arabic numbers from 1 to 100 are presented).
  • the type of elements that will be displayed, as well as the age settings, are controlled by the parent or supervising adult;
  • a Picture Dictionary mode (e.g., Animal Picture Dictionary, Art Gallery Picture Dictionary) which presents individual items, alongside pictures and a written description of said item.
  • a "speaker" button at the side of the description can be clicked to play an audio file containing a verbalized version of the description.
  • users can feel the sensation of the different textures present in it (e.g., Animal Picture Dictionary - sliding finger provides sensation of specific animals' skin texture).
  • 3D models of the subject item will also be available to be explored on a "3D model” viewer ( Figure 6).
  • users can explore the item through normal touch controls, such as rotating two fingers to rotate, or pinching out to zoom in, but they can also slide their fingers on top of the object to feel its textures.
  • This mode is aimed towards both educational and entertainment purposes. This mode is mainly aimed at users above 4 years old;
  • This mode has two submodes, one in which the user is meant to read a story by themselves (Submode A), and another in which a story is to be read to the user, either by another person at their side (e.g., parent) or by the device's text-to-speech function, with audio being transmitted from the device's speakers (Submode B).
  • Submode A the user will receive different haptic feedback sensations, prompted by the story (e.g., "an earthquake could be felt” is accompanied by vibrotactile stimuli representing the phenomenon).
  • a pre-selection of stories is available on the device, with content aimed at different ages.
  • This mode is aimed towards both educational and entertainment purposes. This mode is mainly aimed at users above 5 years old;
  • Submode A User reading the story themselves: user slides finger below and along the sentence they are reading. When the finger hits a certain "trigger” word (e.g., “the ground shook”; “its hair was like silk") the corresponding haptic feedback sensation is generated (e.g., the device vibrates; texture sensation of silky hair is generated until another "trigger' word, indicating the end of this feedback, is reached) (Figure 7);
  • a certain "trigger” word e.g., "the ground shook”; “its hair was like silk”
  • the corresponding haptic feedback sensation is generated (e.g., the device vibrates; texture sensation of silky hair is generated until another "trigger' word, indicating the end of this feedback, is reached) (Figure 7);
  • Submode B User being read the story aloud by another person or the device's text-to-speech function: as with when reading the story themselves, the user should slide their finger below and alongside the sentence that is being read. If another person is reading the story aloud to the user, they should try to either match the user's pace, or lead the user's finger as they are reading. If text-to-speech is being used to read the story aloud, the user's finger sliding speed dictates the pace of the audio, with each word only being read when the finger is slid under it.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

Device for providing an application user interface with haptic feedback for a passenger in an automotive setting, comprising: a haptic surface comprising actuators for exhibiting tactile textures, comprising: a touch sensitive layer, a frictional haptic layer comprising an insulative sheet for user touch and a transparent conductive electrode film for providing frictional haptic sensations, and a vibrotactile haptic layer for providing vibrotactile haptic sensations; a display for displaying images of objects of said application user surface; an electronic data processor configured to: drive the haptic layers to exhibit a tactile texture when the user touches the image of an object in said display; to provide haptic feedback as the user touches the device.

Description

D E S C R I P T I O N
DEVICE AND METHOD FOR PROVIDING AN APPLICATION USER INTERFACE WITH HAPTIC FEEDBACK
TECH N ICAL FI E LD
[0001] The present disclosure relates to a device and method for providing an application user interface with haptic feedback for a passenger in an automotive setting.
BACKG ROU N D
[0002] CN213092557U discloses a sense of touch that can speak turns over voiced book, including the book body, the internal page or leaf in a plurality of books that is provided with of book, all printed the characters pattern on the page or leaf in every book, the book body is including left book body and right book body, left side book body and right book body connect and be the books form, and left book body can be followed left book body and right book body junction and opened and closed, right side book body is equipped with sound generating mechanism by right side edge, sound generating mechanism includes the sound generating mechanism shell and sets up loudspeaker and the PCB board in the sound generating mechanism shell, loudspeaker and PCB board electric connection, be provided with volume on the sound generating mechanism shell and keep off the position switch, be provided with book page or leaf switch in this volume below that keeps off the position switch, page or leaf switch and volume keep off the position switch and all are connected with the PCB board in the book. The utility model discloses simple structure, space utilization is high, and is small, conveniently carries, and page or leaf and sound generating mechanism separately set up in the book simultaneously, cannot have the condition emergence of sound production confusion or non-sound production, and user experience feels good.
[0003] KR20160010843A discloses a method for playing an audio book provided with a vibration function, comprising the steps of: loading sound source data to play the audio book and obtaining total play time of the sound source data loaded; playing the sound source data; loading one or more from vibration data and illustration data corresponding to the sound source data; obtaining one or more from the vibration information and the illustration information corresponding to the play time of the sound source data from one or more from the vibration data and the illustration data; and outputting vibration or illustration based on one or more from the vibration data or the illustration data obtained.
[0004] US2016240102A1 discloses a system for enhanced learning that combines tactile surfaces and audio recordings, allowing the user to explore a 3-dimensional object through touch while also hearing pre-recorded audio explaining regions or features of interest. One of a collection of 3-dimensional tactile modules is used with the system at a given time. When a tactile module is inserted into the system (or otherwise connected) by the user, the system automatically recognizes the inserted module, locates the set of audio recordings and region maps associated with that tactile module, and then plays module-specific audio recordings.
[0005] Document EP2972687A1 discloses a method of receiving user input includes operating a software program in a first operating mode and modifying a graphical interface of the software program in response to receiving gesture. The method also includes receiving a haptic input and operating the software program in a second operating mode in response to the haptic input. In the second operating mode, the gesture produces a different modification to the graphical interface than in the first operating mode.
[0006] These facts are disclosed in order to illustrate the technical problem addressed by the present disclosure.
GENERAL DESCRIPTION
[0007] The present disclosure relates to a device and method for providing an application user interface with haptic feedback for a passenger in an automotive setting. [0008] One of the main ways in which children explore the world around them is through touch. When human physicality is used, especially the sense of touch, the brain receives a higher amount of stimulation. Thus, by physically interacting with "things", children acquire learning experiences that provide important contributions for their development, with a positive impact on their comprehension of the world around them (Shibasaki et al., 2020). Due to this, many museums (e.g., Boston Children's Museum, n.d.; Please Touch Museum, n.d.) have created hands-on exhibition activities, aimed at children, with the goal of creating a greater learning environment regarding the content of their exhibitions. Due to the role of touch on a child's development, research on how haptic feedback sensations can be used to help improve children's education has been performed.
[0009] Studies, such as Yannier et al. (2015), Zhao et al. (2015), and (Shibasaki et al., 2020), indicate that the introduction of haptic feedback sensations on children's reading and listening activities has a positive impact on several aspects, such as the children's ability to comprehend and recall parts of the story, while also enhancing the children's enjoyment of these activities, although these benefits might only be truly effective for children above 4 years old. Other activities, such as drawing and painting, also contribute to a child's ability to recollect events and information, although these effects also seem to be more effective with children above the age of 4 years (Butler et al., 1995). By introducing haptic feedback modalities into drawing activities, richer, more immersive, and more entertaining experiences can be created and presented to children, which can have positive impacts on their development. All these activities, among others, that are enhanced by the presence of haptic feedback sensations, can be either partially or fully implemented as applications (apps) in modern entertainment systems, such as tablets and computers. With current display and surface technologies, devices can be made so that their touchscreen surfaces have the ability to provide users with different tactile feedback sensations beyond those that are normally felt when touching the surface, such as vibrotactile (KHOSHKAVA et al., 2019) and frictional (Olley et al., 2020) stimuli. [0010] Although many edutainment apps, aimed at children, are available on several app markets, the haptic feedback they provide, if any, is strictly vibrotactile, thus existing a need for fully generating the benefits to children's development that arise from the stimulation of the sense of touch.
[0011] In an embodiment, taking into consideration the technological advancements that have recently occurred in the field of surface haptics, it is currently possible to create and develop edutainment apps, aimed at children, that make use of the haptic feedback capabilities with which touchscreen displays and surfaces can now be enhanced with, to provide children with fun, engaging, and educational activities, enhanced by the sensations that haptic feedback can provide. By introducing touchscreen devices and surfaces with haptic feedback capabilities on the interior of vehicles, children can enjoy the activities provided by these apps while riding with adults on their commutes. In the near future, it is expected that level 5 autonomous vehicles will be widely adopted by general consumers. Thanks to these vehicles, adults will have to dedicate less time and attention to the task of driving the vehicle and can instead spend said time and attention bonding with their children, joining in on these edutainment activities, be it as simple play partners, or as tutors, roles that are both helpful and appreciated by the children.
[0012] An aspect of the present disclosure relates to making use of a smart surface, composed of an insulative layer, fitted atop a transparent and conductive electrode sheet layer, which is in turn fitted atop a touch sensitive layer. This touch-sensitive layer is fitted atop a layer of LED lights, which are in turn fitted atop a layer of vibrotactile actuators. The layer of LED lights is optional, and when not presented, the touch-sensitive layer is fitted atop the layer of vibrotactile actuators. This smart surface can be employed either as: a) the interactive surface interface of a display device to which it is physically connect (e.g., the smart surface is part of the device's display); or b) as the interactive surface interface of a display device to which it is not physically connected (e.g., touch surface interface communicating wirelessly with a display device in close proximity). The capacitive-based touch sensitive layer can be made of a variety of materials, depending on the intended use case and aesthetic appeal (e.g., transparent glass - display device; translucent plastic - interactive surface interface for an external device with LED lights underneath it). This smart surface can generate both frictional and vibrotactile haptic sensations. Frictional haptic sensations are generated thanks to the insulative and conductive layers, which allow for friction between the smart surface and the user's skin to be modulated through electrostatic actuation, while the skin is in motion. Vibrotactile haptic sensations, in turn, are generated thanks to the vibrotactile actuators, which can generate vibrations strong enough to be felt through the user's skin, either stationary or in motion, when in direct contact with the smart surface. When the smart surface is used as the interactive interface for an external device, haptic feedback sensations, caused by the external device's software, can be generated on it, as long as the haptic actuation technology that is present on the smart surface is both capable of interpreting the software's information and of generating such sensations.
[0013] Since the present disclosure is mainly aimed at being used by children while they are passengers in a vehicle, and keeping in mind that several health and safety organizations, as well as several legislations worldwide, recommend keeping child passengers on the back seats as much as possible until they are at least 12 years old. Therefore, for the embodiments being discussed in this document, this smart surface is to be fitted on at least one region inside the vehicle: the display surface of an entertainment display on the back of vehicle, where it will serve as said device's interactive surface interface. This device can be attacked to an individual adjustable gooseneck holder, which can be extended and manipulated so that it is placed at a comfortable location for the user to interact with. Other entertainment displays inside the vehicle can also be fitted with said smart surface, such as other entertainment display devices available for rear passengers, but the use of the disclosure should be preferably limited according to the above-mentioned restrictions regarding child passengers.
[0014] It is disclosed a device for providing an application user interface with haptic feedback for a passenger in an automotive setting, comprising: a haptic surface comprising actuators for exhibiting tactile textures, comprising: a touch sensitive layer, a frictional haptic layer comprising an insulative sheet for user touch and a transparent conductive electrode film for providing frictional haptic sensations, and a vibrotactile haptic layer for providing vibrotactile haptic sensations; a display for displaying images of objects of said application user surface; an electronic data processor configured to: drive the haptic layers to exhibit a tactile texture when the user touches the image of an object in said display; to provide haptic feedback as the user touches the device.
[0015] In an embodiment, the touch sensitive layer is a capacitive-based touch sensitive layer.
[0016] In an embodiment, the electronic data processor is further configured to drive the haptic layers to exhibit a tactile texture whose roughness is defined dependent of a colour of the touched object in said display.
[0017] In an embodiment, the electronic data processor is further configured to drive the haptic layers to exhibit a tactile texture whose roughness is defined dependent of the touched object being within or outside a predetermined boundary in said display.
[0018] In an embodiment, the electronic data processor is further configured to drive the haptic layers to exhibit a tactile texture whose roughness is defined dependent of the touched object being along or outside a predetermined path in said display.
[0019] In an embodiment, the electronic data processor is further configured to play an audio file or an audio streaming when the object is touched.
[0020] In an embodiment, the touch sensitive layer is made of translucent plastic comprising LEDs underneath it.
[0021] In an embodiment, the automotive setting is an autonomous vehicle and the display is an entertainment display arranged to be mounted on the back of the vehicle. [0022] In an embodiment comprises an adjustable gooseneck holder or a headrest tablet holder for supporting the device.
[0023] It is also disclosed a method of operating a device according to any of the embodiments, comprising using said electronic data processor for: driving haptic layers to exhibit a tactile texture to provide haptic feedback as the user touches the touchpad device interface.
BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The following figures provide preferred embodiments for illustrating the disclosure and should not be seen as limiting the scope of invention.
[0025] Figure 1 illustrates a schematic illustration of a child user on back seat using app on haptic display device according to an embodiment.
[0026] Figure 2 illustrates a schematic illustration of an application home screen (different waveform indicate different texture sensations) according to an embodiment.
[0027] Figure 3 illustrates a schematic illustration of a drawing with different colours (different waveform indicate different texture sensations) according to an embodiment.
[0028] Figure 4 illustrates a schematic illustration of a drawing inside and outside the lines (different waveform indicate different texture sensations) according to an embodiment.
[0029] Figure 5 illustrates a schematic illustration of a copying a item to learn how to write it (different waveform indicate different texture sensations) according to an embodiment.
[0030] Figure 6 shows a schematic illustration of an animal picture dictionary, exemplary page (different waveform indicate different texture sensations) according to an embodiment. [0031] Figure 7 shows a schematic illustration of a haptic sensations which then triggers from and up to certain "trigger" words (different waveform indicate different texture sensations) according to an embodiment.
DETAILED DESCRIPTION
[0032] The present disclosure relates to a device for providing an application user interface with haptic feedback for a passenger in an automotive setting, comprising: a haptic surface comprising actuators for exhibiting tactile textures, comprising: a touch sensitive layer, a frictional haptic layer comprising an insulative sheet for user touch and a transparent conductive electrode film for providing frictional haptic sensations, and a vibrotactile haptic layer for providing vibrotactile haptic sensations; a display for displaying images of objects of said application user surface; an electronic data processor configured to: drive the haptic layers to exhibit a tactile texture when the user touches the image of an object in said display; to provide haptic feedback as the user touches the device.
[0033] Figure 1 shows a schematic representation of an embodiment of a child user on back seat using app on haptic display device 1 according to an embodiment, for providing a user interface 2.
[0034] In an embodiment, the haptic feedback children edutainment app, on which this document focuses on, can be installed and accessed through several different devices, such as the vehicle's infotainment system or the user's personal smartphone, as long as the software permits it. However, the full benefits of the haptic feedback sensations described below can only be felt when the interaction between user and app is done through a device or surface which possesses actuators capable of generating the intended sensations. Therefore, the interaction between user and app is more fruitful when done through a smart surface as , especially when said interaction occurs inside the vehicle, during the user's commute. The haptic feedback sensations described in this section make use of frictional and/vibrotactile stimuli, which can be generated through electrostatic and vibrotactile actuation, respectively. Although the main innovation of this app is the employment of haptic feedback sensations during the interactions between it and the user, there are no impediments regarding installing and using this app on devices without actuators that can produce the stimuli types from which this app benefits, as long as users keep in mind that the full intended experience cannot be achieved in said devices.
[0035] Regarding activity content of the app itself, these will be described hereafter. These descriptions will focus on use cases of child passengers sitting on the back seats of a vehicle, with access to an individual haptic feedback touch display device, while alongside and not alongside an adult. This application has 5 main activities, accessible from its main menu, Figure 2:
- A free drawing mode in which users are presented with an empty, unrestricted canvas, where they can freely draw and paint with their fingers, using a variety of colours, with each colour giving their own distinct haptic feedback sensation (e.g., haptic feedback ranging from rougher to smoother, depending on the colour's RGB values) (Figure 3). At any time during or after finishing the drawing/painting, the brush tool can be disengaged so that users can freely explore their creations through touch controls and by sliding their fingers across the display surface. At any time during, or after finishing the drawing/painting, the image can be saved in the device's memory to be loaded later or exported through a USB or wireless connection to another device (if available), as not to restrict the user's creation to only being available on the vehicle's displays. This mode is mainly aimed at users around or above 4 years old;
- A figure drawing and painting mode, in which the outline of different items (e.g., animals, objects, fruits) will be displayed, with the user being tasked with painting inside the outlines. In this mode, two extra modes are available, one, "Original", in which the correct colours should be used for the corresponding portion, and another, "Freestyle", in which the user is free to use whatever colour they wish. As in the free drawing mode, each colour will give a corresponding haptic feedback sensation. When painting outside the outline, a rougher haptic sensation is felt to indicate to the user that they are painting out of bounds (Figure 4). As in the free drawing mode, at any time during, or after finishing the drawing/painting, the brush tool can be disengaged so that users can freely explore their creations through touching or sliding their fingers. At any time during, or after finishing the drawing/painting, the image can be saved in the device's memory to be loaded later, or exported through a USB or wireless connection to another device, to not restrict the user's creation to only the in-vehicle display. This mode is mainly aimed at users around or above 4 years old;
- A follow-along, in which "flashcards" containing the visualization of different elements (e.g., roman numerals, Arabic numerals, Latin alphabet letters, Hiragana, Kanjis) are displayed on either the right or the left side of the screen, one at a time, alongside an audio file containing the correct vocalization of said element, which is played when the item is loaded in. The user's task, in this mode, is to draw the visualization of the element present on the screen on the other portion of the screen not displaying the element (which side is the visualization and the drawing portion of the screen can be adjusted in settings by the parent or supervising adult, depending on if the user is left-handed, righthanded, or ambidextrous). While following along correctly, a smoother haptic sensation is felt. When following along incorrectly, a rougher haptic sensation is felt instead. The user can also slide their finger on the visualization portion of the screen, feeling a smooth texture sensation when sliding on top of the line of the element, and a rougher texture sensation when sliding outside the lines of the element (Figure 5). This mode is aimed more towards educational purposes. This mode is mainly aimed at users above 4 years old, with different settings available depending on the user's age (e.g., 4-year-olds - all Latin alphabet letters are displayed in their uppercase form, by themselves, and Arabic numbers from 1 to 10 are presented; 6-year-olds - Latin alphabet letters can be displaying in uppercase and lowercase form, alongside other letters to form words, and Arabic numbers from 1 to 100 are presented). The type of elements that will be displayed, as well as the age settings, are controlled by the parent or supervising adult;
- A Picture Dictionary mode (e.g., Animal Picture Dictionary, Art Gallery Picture Dictionary) which presents individual items, alongside pictures and a written description of said item. A "speaker" button at the side of the description can be clicked to play an audio file containing a verbalized version of the description. When sliding their finger across the picture, users can feel the sensation of the different textures present in it (e.g., Animal Picture Dictionary - sliding finger provides sensation of specific animals' skin texture). Whenever possible, 3D models of the subject item will also be available to be explored on a "3D model" viewer (Figure 6). In this viewer, users can explore the item through normal touch controls, such as rotating two fingers to rotate, or pinching out to zoom in, but they can also slide their fingers on top of the object to feel its textures. This mode is aimed towards both educational and entertainment purposes. This mode is mainly aimed at users above 4 years old;
- Haptic story time mode. This mode has two submodes, one in which the user is meant to read a story by themselves (Submode A), and another in which a story is to be read to the user, either by another person at their side (e.g., parent) or by the device's text-to-speech function, with audio being transmitted from the device's speakers (Submode B). At certain, specific parts of each story, the user will receive different haptic feedback sensations, prompted by the story (e.g., "an earthquake could be felt" is accompanied by vibrotactile stimuli representing the phenomenon). A pre-selection of stories is available on the device, with content aimed at different ages. This mode is aimed towards both educational and entertainment purposes. This mode is mainly aimed at users above 5 years old;
Submode A - User reading the story themselves: user slides finger below and along the sentence they are reading. When the finger hits a certain "trigger" word (e.g., "the ground shook"; "its hair was like silk") the corresponding haptic feedback sensation is generated (e.g., the device vibrates; texture sensation of silky hair is generated until another "trigger' word, indicating the end of this feedback, is reached) (Figure 7);
Submode B - User being read the story aloud by another person or the device's text-to-speech function: as with when reading the story themselves, the user should slide their finger below and alongside the sentence that is being read. If another person is reading the story aloud to the user, they should try to either match the user's pace, or lead the user's finger as they are reading. If text-to-speech is being used to read the story aloud, the user's finger sliding speed dictates the pace of the audio, with each word only being read when the finger is slid under it.
[0036] The term "comprising" whenever used in this document is intended to indicate the presence of stated features, integers, steps, components, but not to preclude the presence or addition of one or more other features, integers, steps, components, or groups thereof.
[0037] The disclosure should not be seen in any way restricted to the embodiments described and a person with ordinary skill in the art will foresee many possibilities to modifications thereof. The above-described embodiments are combinable. The following claims further set out particular embodiments of the disclosure.
[0038] References
1. Boston Children's Museum, (n.d.). Boston Children's Museum Home Page. Retrieved January 12, 2021.
2. Butler, S., Gross, J., & Hayne, H. (1995). The effect of drawing on memory performance in young children. Developmental Psychology, 31(4), 597-608.
3. KHOSHKAVA, V., CRUZ-HERNANDEZ, J. M., & SHAH, K. (2019). HAPTIC ACTUATOR ASSEMBLY WITH A SPRING PRE-LOAD DEVICE (Patent No. EP3582074A1).
4. Olley, M. F. D., Peshkin, M. A., & Colgate, J. E. (2020). Electronic controller haptic display with simultaneous sensing and actuation (Patent No. US10768749B2). Please Touch Museum, (n.d.). Please Touch Museum Home Page. Retrieved January 12, 2021, from https://www.pleasetouchmuseum.org Shibasaki, M., Kamiyama, Y., Czech, E., Obata, K., Wakamoto, Y., Kishi, K., Hasegawa, T., Tsuchiya, S., Matsuda, S., & Minamizawa, K. (2020). Interest Arousal by Haptic Feedback During a Storytelling for Kindergarten Children. In I. Nisky, J. Hartcher-O'Brien, M. Wiertlewski, & J. Smeets (Eds.), Haptics: Science, Technology, Applications (pp. 518-526). Springer International Publishing. Yannier, N., Israr, A., Lehman, J. F., & Klatzky, R. L. (2015). FeelSleeve: Haptic Feedback to Enhance Early Reading. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 2015- April, 1015- 1024. Zhao, S., Lehman, J., Israr, A., & Klatzky, R. (2015). Using haptic inputs to enrich story listening for young children. Proceedings of the 14th International Conference on Interaction Design and Children, 239-242.

Claims

C L A I M S Device (1) for providing an application user interface (2) with haptic feedback for a passenger in an automotive setting, comprising: a haptic surface comprising actuators for exhibiting tactile textures, comprising: a touch sensitive layer, a frictional haptic layer comprising an insulative sheet for user touch and a transparent conductive electrode film for providing frictional haptic sensations, and a vibrotactile haptic layer for providing vibrotactile haptic sensations; a display for displaying images of objects of said application user surface; an electronic data processor configured to: drive the haptic layers to exhibit a tactile texture when the user touches the image of an object in said display; to provide haptic feedback as the user touches the device. Device (1) for providing an application user interface (2) with haptic feedback according to any of the previous claims wherein the touch sensitive layer is a capacitive-based touch sensitive layer. Device (1) for providing an application user interface (2) with haptic feedback according to any of the previous claims wherein the electronic data processor is further configured to drive the haptic layers to exhibit a tactile texture whose roughness is defined dependent of a colour of the touched object in said display. Device (1) for providing an application user interface (2) with haptic feedback according to any of the previous claims wherein the electronic data processor is further configured to drive the haptic layers to exhibit a tactile texture whose roughness is defined dependent of the touched object being within or outside a predetermined boundary in said display. Device (1) for providing an application user interface (2) with haptic feedback according to any of the previous claims wherein the electronic data processor is further configured to drive the haptic layers to exhibit a tactile texture whose roughness is defined dependent of the touched object being along or outside a predetermined path in said display. Device (1) for providing an application user interface (2) with haptic feedback according to any of the previous claims wherein the electronic data processor is further configured to play an audio file or an audio streaming when the object is touched. Device (1) for providing an application user interface (2) with haptic feedback according to any of the previous claims wherein the touch sensitive layer is made of translucent plastic comprising LEDs underneath it. Device (1) for providing an application user interface (2) with haptic feedback according to any of the previous claims wherein the automotive setting is an autonomous vehicle and the display is an entertainment display arranged to be mounted on the back of the vehicle. Device (1) for providing an application user interface (2) with haptic feedback according to the previous claim comprising an adjustable gooseneck holder or a headrest tablet holder for supporting the device. Method of operating a device (1) according to any of the claims 1-9, comprising using said electronic data processor for: driving haptic layers to exhibit a tactile texture to provide haptic feedback as the user touches the touchpad device interface.
EP21854756.0A 2021-12-27 2021-12-28 Device and method for providing an application user interface with haptic feedback Pending EP4457587A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
PT11769121 2021-12-27
PCT/IB2021/062393 WO2023126648A1 (en) 2021-12-27 2021-12-28 Device and method for providing an application user interface with haptic feedback

Publications (1)

Publication Number Publication Date
EP4457587A1 true EP4457587A1 (en) 2024-11-06

Family

ID=80218743

Family Applications (1)

Application Number Title Priority Date Filing Date
EP21854756.0A Pending EP4457587A1 (en) 2021-12-27 2021-12-28 Device and method for providing an application user interface with haptic feedback

Country Status (2)

Country Link
EP (1) EP4457587A1 (en)
WO (1) WO2023126648A1 (en)

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101895818B1 (en) * 2012-04-10 2018-09-10 삼성전자 주식회사 Method and apparatus for providing feedback associated with e-book in terminal
WO2015127257A1 (en) 2014-02-21 2015-08-27 Northwestern University Haptic display with simultaneous sensing and actuation
US9261908B2 (en) 2013-03-13 2016-02-16 Robert Bosch Gmbh System and method for transitioning between operational modes of an in-vehicle device using gestures
JP2015130168A (en) * 2013-12-31 2015-07-16 イマージョン コーポレーションＩｍｍｅｒｓｉｏｎ Ｃｏｒｐｏｒａｔｉｏｎ Friction augmented control, and method to convert buttons of touch control panels to friction augmented controls
KR20160010843A (en) 2014-07-17 2016-01-28 (주)북큐브네트웍스 Method for playing audio book with vibration, device and computer readable medium
US20160240102A1 (en) 2015-02-12 2016-08-18 Vikram BARR System for audio-tactile learning with reloadable 3-dimensional modules
US10345910B1 (en) 2018-06-15 2019-07-09 Immersion Corporation Haptic actuator assembly with a spring pre-load device
CN111026275B (en) * 2019-12-12 2021-02-26 深圳市华星光电半导体显示技术有限公司 Electrostatic feedback display array and active driving method and circuit
CN213092557U (en) 2020-08-31 2021-04-30 深圳市童儿乐文化科技有限公司 Touch turning vocal book capable of speaking

Also Published As

Publication number Publication date
WO2023126648A1 (en) 2023-07-06

Similar Documents

Publication Publication Date Title
JP4225567B2 (en) Spatial access method for time-based information
US6920619B1 (en) User interface for removing an object from a display
AU2004214467B2 (en) Hand-held interactive electronic device
CN107077229B (en) Human-machine interface device and system
US20210056866A1 (en) Portable Reading, Multi-sensory Scan and Vehicle-generated Motion Input
Brivio et al. A soft, mobile, autonomous robot to develop skills through play in autistic children
Lévesque et al. Adaptive level of detail in dynamic, refreshable tactile graphics
KR101550346B1 (en) Method of Reproducing Content-App based Picture Book Contents for Prenatal Education for Pregnant Women in Multi-cultural Families
Daniels Children’s engagement with iPads in early years classrooms: Exploring peer cultures and transforming practices
Bornman et al. Background, features, and principles of AAC technology
EP4457587A1 (en) Device and method for providing an application user interface with haptic feedback
Brett Adaptive technology for special human needs
Glos et al. Rosebud: a place for interaction between memory, story, and self
Cicconi Augmented total theatre: shaping the future of immersive augmented reality representations
Søyland et al. Sense-making through touch interaction with a picturebook app
KR101535457B1 (en) Multi-function toy for studying and playing
Barrass ZiZi: The Affectionate Couch and the Interactive Affect Design Diagram
Montemayor Physical programming: tools for kindergarten children to author physical interactive environments
Fakhri et al. Haptics for sensory substitution
Kunjam Tactile narratives in virtual reality
Archambault et al. Design of augmented tactile books for blind children
Avendano et al. ReCapture: A VR interactive narrative experience concerning perspectives and self-reflection
Bell Participatory art and computers: identifying, analysing and composing the characteristics of works of participatory art that use computer technology
Weinberg et al. The baby sense environment: enriching and monitoring infants' experiences and communication
Angel et al. Memory and motion

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20240627

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20260310