WO2018130273A1 - Displaying text to a user of a computing device - Google Patents

Displaying text to a user of a computing device Download PDF

Info

Publication number
WO2018130273A1
WO2018130273A1 PCT/EP2017/050452 EP2017050452W WO2018130273A1 WO 2018130273 A1 WO2018130273 A1 WO 2018130273A1 EP 2017050452 W EP2017050452 W EP 2017050452W WO 2018130273 A1 WO2018130273 A1 WO 2018130273A1
Authority
WO
WIPO (PCT)
Prior art keywords
text
presentation
user
reading
computing device
Prior art date
Application number
PCT/EP2017/050452
Other languages
French (fr)
Inventor
Jan Jasper VAN DEN BERG
Matthew John LAWRENSON
Original Assignee
Telefonaktiebolaget Lm Ericsson (Publ)
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Telefonaktiebolaget Lm Ericsson (Publ) filed Critical Telefonaktiebolaget Lm Ericsson (Publ)
Priority to PCT/EP2017/050452 priority Critical patent/WO2018130273A1/en
Publication of WO2018130273A1 publication Critical patent/WO2018130273A1/en

Links

Classifications

    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B19/00Teaching not covered by other main groups of this subclass
    • G09B19/04Speaking
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00Handling natural language data
    • G06F40/10Text processing
    • G06F40/103Formatting, i.e. changing of presentation of documents
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B19/00Teaching not covered by other main groups of this subclass
    • G09B19/06Foreign languages
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/24Speech recognition using non-acoustical features
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G10L25/51Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination

Definitions

  • the invention relates to a computing device for displaying a text to a user of the computing device, a method of displaying a text to a user of a computing device, a corresponding computer program, and a corresponding computer program product.
  • the user's context relates to properties of the computing device which is being used for displaying content, e.g., a size and/or resolution of the display, or an available bandwidth over a wireless connection, e.g., via a cellular mobile communications network or a Wireless Local Area Network (WLAN)/Wi-Fi network.
  • the usage semantics of the content relates to the task for which the content is being used, and is typically user specific ("URICA: Usage-awaRe Interactive Content Adaptation for Mobile Devices", by I. Mohomed, J. C. Cai, and E.
  • adaptation of content for display is dependent on the user viewing the content as well as a task the user is performing, such as scanning through a text for finding relevant information, quickly reading a text for grasping a theme of the text, or carefully reading a text for learning.
  • a computing device for displaying a text to a user of the computing device.
  • the computing device comprises processing means which is operative to display a presentation of the text, acquire a representation of the user subvocalizing at least a part of the presentation of the text, and measure one or more reading parameters from the acquired representation. The reading
  • the processing means is further operative to adapt the displayed presentation of the text based on the measured reading parameters.
  • a method of displaying a text to a user of a computing device is provided.
  • the method is performed by the computing device and comprises displaying a presentation of the text, acquiring a representation of the user subvocalizing at least a part of the presentation of the text, and measuring one or more reading parameters from the acquired representation.
  • the reading parameters characterize a manner in which the user is reading the presentation of the text.
  • the method further comprises adapting the displayed presentation of the text based on the measured reading parameters.
  • a computer program is provided.
  • the computer program comprises computer-executable
  • a computer program product comprises a computer- readable storage medium which has the computer program according to the third aspect of the invention embodied therein.
  • the invention makes use of an understanding that an improved adaption of content, in particular text, to a user of a computing device may be achieved based on subvocalization.
  • Subvocalization is the "internal speech" while reading a text and is characterized by minute movements of the larynx and other speech muscles, as well as other body parts, which are imperceptible to the human eye but detectable by technical means.
  • the displayed presentation of the text may be adapted based on the measured reading parameters.
  • a presentation of a text is to be understood as to encompass the information contained in the text, or a subset thereof, as well as the way it is presented to the user, i.e., displayed.
  • different presentations of a text may, e.g., be the complete text, one or more consecutive parts of the text, e.g., sentences, or a summary of the text.
  • different presentations of the text may, e.g., utilize different layouts, fonts, styles, and magnification.
  • the one or more reading parameters may, e.g., be a reading speed, a reading continuity, a subvocal activity, or a combination thereof.
  • one of the reading parameters may also reflect whether the user is reading the text in the order it is written, or "hopping" back and forth when reading. The latter may, e.g., indicate that the user is searching for information.
  • Embodiments of the invention are advantageous since people increasingly prefer viewing content, such as text and images, which are displayed by a computing device, either on a built-in screen or an external screen, rather than a printed paper copy.
  • the computing device may, e.g., be a mobile phone, a mobile terminal, a smartphone, a tablet, a laptop, an e- book reader, a television, or a media player, which typically are provided with built-in screens, oftentimes of touchscreen type.
  • the computing device may be a personal computer which is operatively connected to an external a computer display.
  • embodiments of the invention enable an improved adaption of text displayed to a user based on measured reading parameters which reflect the manner in which the user is reading the text.
  • the user's manner in reading the text is related to a task the user is performing, such as scanning through a text for finding relevant information, quickly reading through a text for grasping a theme gist of the text, or carefully reading through a text for learning, but may also be related to the user's attitude towards the text, i.e., the user's comprehension of the text, his/her interest in the text of the information contained therein, and the user's focus when reading a presentation of the text.
  • the presentation of the text is adapted so as improve the reading experience of the user and/or to better support the purpose of the text, such as providing an overview over a subject, providing detailed information about a subject, or gaining the user's interest in a subject.
  • the representation of the user subvocalizing at least a part of the presentation of the text may be acquired by acquiring nerve signals which are captured from a throat of the user subvocalizing the part of the presentation of the text, and deriving the representation of the user subvocalizing at least a part of the presentation of the text as a representation of the nerve signals.
  • the nerve signals are acquired from one or more sensors, or electrodes, which are operatively connected to the computing device, in particular electromyographical (EMG) sensors or electrodes.
  • EMG electromyographical
  • Both the nerve signals and the resulting motions are correlated with the subvocalized text and can be detected.
  • a single subvocal movement has strong correlations to a single syllable, and the number of syllables per unit of time, which is a measure of reading speed, can be detected by measuring the speed of subsequent subvocal movements.
  • the subvocal movements follow each other with a speed above a certain threshold, this indicates that they belong to the same word.
  • the detected subvocal activity is below a certain threshold, it can be concluded that the user makes a pause since he or she has reached a punctuation mark in the text, or because he/she is not reading.
  • These internal and external motions include muscle movements which are related to the ones the person would make to audibly create the sounds, such as the movement of vocal chords, and may also include associated movements the person would make, e.g., sway due to a rhythm of the text and modified breathing patterns. Examples of these motions are motions of vocal chords, lips, tongue, jaw, neck, other body parts such as heads and shoulders, and motions related to breathing.
  • the representation of the user subvocalizing at least a part of the presentation of the text may be acquired by acquiring a video sequence of the user subvocalizing the part of the presentation of the text, magnifying motions of one or more body parts of the user, and deriving the representation of the user subvocalizing at least a part of the presentation of the text as a representation of the magnified motions.
  • the motions of the one or more body parts are correlated with the subvocalized part of the presentation of the text.
  • the representation of the magnified motions may, e.g., be derived using one or more metrics
  • the video sequence is acquired from a camera operatively connected to the computing device, e.g., a camera which is comprised in the computing device, such as a front-facing camera in a smartphone or tablet.
  • a camera which is comprised in the computing device, such as a front-facing camera in a smartphone or tablet.
  • an external camera such as a web cam which is connected to the computing device via an interface may be used.
  • the camera captures motions of the one or more body parts of the user, such as the user's lips, throat, mouth, nose, face, neck, or shoulders.
  • the motions may, e.g., be magnified by means of Eulerian Video Magnification (EVM) or Lagrangian techniques which are described in further detail below.
  • EVM Eulerian Video Magnification
  • Lagrangian techniques which are described in further detail below.
  • Acquiring the representation of at least a part of the presentation of the text subvocalized by the user by means of a video capturing the user subvocalizing when reading, and processing the video to magnify motions which are correlated with the subvocalized text, is an alternative to utilizing EMG sensors for capturing nerve signals.
  • the displayed image is displayed
  • presentation of the text is adapted based on a comparison of the measured reading parameters and one or more corresponding reference reading parameters. For instance, if reading speed if measured as one of the reading parameters, the measured reading speed may be compared to a
  • reading continuity is measured as another of the reading parameters, indicating the fraction of displayed syllables or words which are actually read and
  • each reference reading parameter may comprise one or more threshold values and/or one or more ranges for defining conditions which trigger adaption of a displayed presentation of a text based on the subvocalization of the user when reading the displayed presentation. As an example, if he measured reading speed is above a certain reading-speed threshold value and the measured reading continuity is below a certain reading-continuity threshold value, it may be concluded that the user is reading hastily through the displayed presentation of the text and is likely to prefer a less detailed presentation of the text.
  • the one or more reference reading parameters may be associatively stored with the text.
  • the one or more reference reading parameters may be stored as meta data, together with the text in an electronic document, or in a separate database.
  • the author or publisher of a text can define how the text is to be read.
  • the one or more reference reading parameters may be derived from acquired representations of the user subvocalizing text by measuring one or more reading parameters from the acquired representations and storing the measured reading parameters as the reference reading parameters.
  • the reference reading parameters are derived in a similar manner as the reading parameters based on which the displayed presentation of the text is adapted.
  • the one or more reference reading parameters are derived from one or more texts which are displayed to, and read by, the user during a training or learning phase, and may optionally be derived during usage of the computing device.
  • the one or more reference reading parameters may be configured by the user, a manufacturer of the device, or provided together with the text.
  • the one or more reference reading parameters may be dependent on a type of the text. This is advantageous since different types of texts are typically read in a different manner. For instance, technical documents or reports are typically read with attention to detail, at rather low speed and with great continuity. News on social networks, on the other hand, are oftentimes read with less attention to detail, and a correspondingly larger reading speed and lower reading continuity, respectively.
  • the displayed image is displayed
  • presentation of the text is adapted further based on a type of the text.
  • This is advantageous since different types of texts have different reading styles, i.e., the manner in which a user is reading a text, associated with it. For instance, a report may be read with attention to detail by one user, characterized by relatively low reading speed and relatively high reading continuity, whereas it may be read with less attention to detail by a different user which is interested in the gist of the text.
  • the displayed image is displayed
  • presentation of the text may be adapted by adding information from the text to the presentation of the text, or by removing information from the displayed presentation of the text. Moreover, information from the text may be added in one part of the presentation and information may be removed in another part. In other words, the level of detail which is presented to the user may be adjusted based on the measured reading parameters. For instance, if the manner in which the user is reading and subvocalizing the displayed presentation of the text is characterized by a relatively low reading speed and a relatively high reading continuity, it may be concluded that the user reads the presented text with attention to detail, and the displayed presentation of the text may be adapted by adding additional information, e.g., additional passages providing a higher level of detail.
  • the manner in which the user is reading and subvocalizing the displayed presentation of the text is characterized by a relatively high reading speed and a relatively low reading continuity, it may be concluded that the user reads the presented text with little attention to detail and is likely attempting to extract the gist of the text.
  • the displayed presentation of the text may be adapted by removing information which the user most likely is not interested in, e.g., passages providing a higher level of detail.
  • the displayed image is displayed
  • presentation of the text may be adapted by providing a summary of at least a part of the text. This is advantageous if the manner in which the user is reading and subvocalizing the presentation of the text is characterized by a relatively high reading speed and a relatively low reading continuity, indicating that the user reads the presented text with little attention to detail and is likely attempting to extract the gist of the text.
  • the displayed presentation of the text may be amended, or replaced, by a summary of the text.
  • the summary may either be provided with the text, e.g., as metadata or as a separate section in an electronic document, such as a file, together with the text, or generated using Natural Language Processing (NLP) techniques.
  • NLP Natural Language Processing
  • the displayed image is displayed
  • presentation of the text may be adapted by increasing a font size of the displayed text and/or magnifying the displayed text. This may be
  • the measured reading parameters indicate an unusually low reading speed, which may be an indication that the user has difficulties deciphering the displayed presentation of the text.
  • the displayed image is displayed
  • presentation of the text may be adapted by adding one or more references to additional information, such as hyperlinks or tool tips.
  • additional information such as hyperlinks or tool tips.
  • the displayed presentation may be adapted by reducing the level of detail, i.e., providing less information in the displayed presentation but adding hyperlinks or tooltips for giving the user the opportunity to access additional information.
  • the displayed presentation of the text may be adapted by changing a layout of the presented text. This is advantageous in situations when the measured reading parameters indicate that the user is hopping back and forth in the displayed presentation, attempting to find specific information. By presenting the parts of the text in different order, the reading experience of the user may be improved.
  • Fig. 1 shows a computing device for displaying a text to a user of the computing device, in accordance with an embodiment of the invention.
  • Fig. 2 shows a computing device for displaying a text to a user of the computing device, in accordance with another embodiment of the invention.
  • Fig. 3 exemplifies an image of a video sequence captured by a camera comprised in the embodiment of the computing device shown in Fig. 2.
  • Fig. 4 illustrates an example text, an expected phrasing when read by a user, and a representation of the expected phrasing as a time-dependent function.
  • Figs. 5A and 5B illustrate adapting a displayed presentation of a text by adding or removing information, in accordance with an embodiment of the invention.
  • Fig. 6 illustrates adapting a displayed presentation of a text by providing a summary of at least a part of the text, in accordance with another embodiment of the invention.
  • Fig. 7 illustrates adapting a displayed presentation of a text by increasing a font size of the displayed text, in accordance with a further embodiment of the invention.
  • Fig. 8 illustrates adapting a displayed presentation of a text by adding one or more references to additional information, in accordance with yet another embodiment of the invention.
  • Fig. 9 shows an embodiment of the processing means comprised in the computing device for displaying a text to a user.
  • Fig. 10 shows another embodiment of the processing means comprised in the computing device for displaying a text to a user.
  • Fig. 1 1 shows a method of displaying a text to a user of the computing device, in accordance with embodiments of the invention.
  • the embodiments of the invention described herein make use of an understanding that text which is displayed to a user of a computing device may be adapted based on detecting, using an acquired representation of a subvocalization by the user, to what extent and/or how the user actually is reading displayed text. This is achieved by measuring one or more reading parameters which characterize a manner in which the user is reading the presentation of the text.
  • the displayed presentation of the text can advantageously be improved so as to better fit the reading style of the user, the task at hand, and/or the intended purpose of the text.
  • the user may be scanning through a text for finding relevant information, quickly reading a text for grasping a theme of the text, or carefully reading a text for learning.
  • the intended purpose of a text may, e.g., be to provide an overview over a subject, to provide detailed information about a subject, or to gain the user's interest in a subject.
  • Subvocalization is the "internal speech" while reading a text and is characterized by minute movements of the larynx and other speech muscles, as well as other body parts, which are imperceptible to the human eye but detectable by technical means.
  • a person reads a text, although they do not make audible noise, they still make a variety of internal (e.g., vocal chords) and external (e.g., lips) motions which can be detected, albeit being imperceptible to the human eye.
  • These motions include muscle movements which are related to the ones the person typically would make to create audible sounds, and which consequently are correlated with the text subvocalized by the user.
  • the movements include: - Lip motions: Each time a person says a syllable their lips move. The pronunciation of each syllable requires the user to move their lips. For instance, pronouncing the word “sen-tence” requires to first pronounce “sen” and then again to pronounce “tence”. - Throat movement: The utterance of a syllable also requires the user to change the shape of their vocal chords. This is done by moving the muscles that control the vocal chords, and this movement in turn moves the surface of the user's throat.
  • - Pauses A person who is reading pauses in between words, and also at punctuation points such as comma, question mark, and full stop. The person may also breath during a pause, resulting in modified breathing patterns.
  • a person who is reading may also sway to the text he/she is reading, in particular if the text is lyrics of a piece of music or poetry.
  • FIG. 1 an embodiment 100 of the computing device for displaying a text to a user 1 10 of computing device 100 is shown.
  • Computing device 100 is operative to adapt a displayed presentation 131 of the text based on one or more reading parameters which are measured from an acquired
  • the one or more reading parameters characterize a manner in which user 1 10 is reading presentation 131 and may, e.g., be a reading speed, a reading continuity, or a combination thereof.
  • Computing device 100 is in
  • Fig. 1 illustrated as a tablet comprising a display 101 , e.g., a touchscreen, a front-facing camera 102, processing means 103, a communications module 104, and an optional interface 105 for connecting external devices, as is described further below.
  • Communications module 104 is operative to effect wireless
  • communications module 104 may be operative to effect wireless communications through a Wireless Local Arena Network (WLAN)/Wi-Fi network.
  • WLAN Wireless Local Arena Network
  • Interface 105 for connecting external devices is in Fig. 1 illustrated as a conventional headphone jack, or headset jack, but may alternatively be embodiment as any other type of connector, such as a Lightning connector, a Universal Serial Bus (USB) connector, or the like. As yet a further alternative, interface 105 may also be a wireless interface based on technologies like coded visible or invisible light, Bluetooth, or ZigBee.
  • Computing device 100 is operative to display presentation 131 of the text to user 1 10, either using built-in display 101 or an external display which is operatively connected to computing device 100, e.g., a computer display.
  • Computing device 100 is further operative to acquire a representation of user 1 10 subvocalizing at least a part of presentation 131 , and to measure one or more reading parameters from the acquired representation. The measured one or more reading parameters characterize a manner in which user 1 10 is reading presentation 131 .
  • Computing device 100 is further operative to adapt presentation 131 text based on the measured reading parameters.
  • computing device 100 is operative to acquire the representation of user 1 10 subvocalizing at least a part of presentation 131 by acquiring nerve signals which are captured from the throat 1 1 1 of user 1 10 when subvocalizing at least a part of
  • the nerve signals may be acquired by means of one or more EMG sensors 120, or electrodes, which are attached to throat 1 1 1 , e.g., one sensor 120 on each side of the larynx, and which are operatively connected to computing device 100, e.g., via interface 105.
  • the captured nerve signals are correlated with the part of presentation 131 which user 1 10 subvocalizes when reading, since they control movements of the larynx and other body parts 1 1 1-1 15 of user 1 10.
  • the nerve signals which are acquired from sensors 120 may either be received directly, e.g., as an analog electric signal, as a digitized representation of the analog sensor signal, or as a chunk of data captured by sensors 120 and delivered to computing device 100, optionally after pre-processing, either wired or wirelessly via interface 105.
  • the representation of user 1 10 subvocalizing at least a part of presentation 131 is derived as a representation of the nerve signals, and can subsequently be used for measuring the one or more reading parameters, as is described further below with reference to Fig. 4.
  • computing device 200 comprises a display 101 , e.g., a touchscreen, a front-facing camera 102, processing means 103, a communications module 104, and an optional interface 105 for connecting external devices.
  • display 101 e.g., a touchscreen
  • processing means 103 e.g., a front-facing camera 102
  • communications module 104 e.g., a communications module 104
  • optional interface 105 for connecting external devices.
  • computing device 200 is operative to display presentation 131 of the text to user 1 10, either using built-in display 101 or an external display which is operatively connected to computing device 200, e.g., a computer display.
  • Computing device 200 is further operative to acquire a representation of user 1 10 subvocalizing at least a part of presentation 131 , and to measure one or more reading parameters from the acquired representation. The measured one or more reading parameters characterize a manner in which user 1 10 is reading presentation 131 .
  • Computing device 200 is further operative to adapt presentation 131 text based on the measured reading parameters.
  • computing device 200 is operative to acquire the representation of user 1 10
  • subvocalizing at least a part of presentation 131 by acquiring a video sequence of user 1 10 subvocalizing at least a part of presentation 131 , and magnifying motions of one or more body parts 1 1 1-1 15 of user 1 10 which are correlated with the subvocalized part of presentation 131 .
  • the video sequence is acquired from a camera operatively connected to computing device 200, such as front-facing camera 102 or an external camera
  • computing device 200 e.g., a web cam or a camera mounted in a computer display.
  • the camera has a field-of-view 130 so as to capture at least one of the user's 1 10 face, throat 1 1 1 , mouth or lips 1 12, nose 1 13, neck 1 14, and shoulders 1 15.
  • An example of an image of a video sequence captured by camera 102 is illustrated in Fig. 3.
  • computing device 200 is operative to magnify the motions of one or more body parts 1 1 1-1 15 by video processing the acquired video sequence, either using EVM or by utilizing Lagrangian techniques.
  • EVM is an image processing technique which applies a spatial decomposition followed by a temporal filter to the frames of a standard video in order to amplify very small motions which are present but typically are imperceptible to the human eye.
  • the technique can be applied in real-time to highlight events occurring at specified temporal frequencies.
  • EVM EVM Magnification for Revealing Subtle Changes in the World
  • computing device 200 is operative to derive the representation of user 1 10 subvocalizing at least a part of presentation 131 as a representation of the magnified motions.
  • the representation may be based on one or more metrics which quantify the motions of lips 1 12, neck 1 14, or other body parts, of user 1 10.
  • the representation may be based on a measured distance 301 , magnified by video processing, between the upper and lower lip 1 12 over time.
  • the representation may be based on a measured width 302, magnified by video processing, of neck 1 14 over time.
  • the representation of user 1 10 subvocalizing the part of first text 131 can subsequently be used for measuring the one or more reading parameters, as is described in the following.
  • a text 401 is illustrated as an example for presentation 131 .
  • text 401 can be represented as a phrasing
  • graph 402 which graphically illustrates the phrasing which user 1 10 applies when reading text 401 .
  • phrasing graph 402 illustrates the duration of each syllable (indicated by “_”), the separation between syllables (indicated by 7"), which is an indication for user 1 10's reading cadence, as well as the duration of pauses (indicated by " ⁇ ") at the commas and the question mark, respectively.
  • phrasing graph 402 is a graphical representation of user 1 10's phrasing when reading text 401
  • user 1 10's phrasing may alternatively be cast into a mathematical representation which is suitable for a more quantitative comparison with the acquired representation of user 1 10 subvocalizing at least a part of presentation 131 .
  • the phrasing for text 401 may be represented by a time-dependent function 403, or a signal, which assumes a value of one ("1 ") during utterance of a syllable by user 1 10, and a value of zero ("0") during a pause between syllables, a pause between words, or a pause due to a punctuation mark.
  • numerous mathematical representations of the phrasing graph for text 401 may be envisaged, and the embodiments of the invention are not limited to the particular form 403 illustrated in Fig. 4.
  • Computing device 100 or 200 is operative to measure the one or more reading parameters from the acquired representation of user 1 10
  • a reading speed which may, e.g., be defined as the number of subvocalized syllables per unit time, may be measured by identifying syllables and pauses in the acquired representation and counting the number of identified syllables during a predetermined time interval, typically a few seconds.
  • Syllables and pauses may be identified by comparing the acquired representation of user 1 10 subvocalizing text with an expected phrasing for the displayed text.
  • the expected phrasing may be derived based on information which is acquired during a training or learning phase. More specifically, a text is displayed to user 1 10, and a representation of user 1 10 subvocalizing the displayed text is acquired.
  • characteristic features may be identified in the acquired representation which coincide with, e.g., the onset of a syllable, the onset of a pause between syllables or words, the termination of a syllable, the termination of a pause between syllables or words, and so forth.
  • the comparison between the acquired representation and the expected phrasing may be performed by deriving a time-dependent function, or signal, for the displayed text, such as function 403 for text 401 , by utilizing predetermined values for the duration of syllables, pauses between syllables, pauses between words, and pauses in relation to punctuation marks, respectively.
  • These predetermined values may, e.g., be configured by a manufacturer of computing device 100 or 200, and subsequently be improved during the training or learning phase by varying the values and calculating a correlation between the thereby obtained time-dependent function, which represents an expected phrasing for the displayed text, and the acquired representation of user 1 10 subvocalizing the displayed text.
  • the respective values for the different durations are then varied until the calculated correlation assumes a maximum, indicating that the corresponding values for the duration of syllables, pauses between syllables, pauses between words, and pauses in relation to punctuation marks, respectively, are the best representation of user 1 10's manner of reading the displayed text, reflecting user-specific properties such as user 1 10's cadence and/or reading speed.
  • correlation is a statistical relationship which reflects the extent to which two random variables, e.g., two time-dependent functions or signals such as a time-dependent function representing an expected phrasing by user 1 10 when subvocalizing a displayed text and the acquired representation of user 1 10 subvocalizing the displayed text, overlap.
  • the correlation between two random variables is commonly referred to as cross-correlation and can be quantified by means of a correlation function, which can be expressed as an integral over the two random variables over time.
  • correlation functions are normalized such that a perfect correlation between the two random variables, i.e., the two random variables are identical, result in a maximum value which oftentimes is chosen to be equal to one ("1 ").
  • the correlation of two completely independent random variables yields a correlation value of zero ("0").
  • An example is the well-known Pearson product-moment correlation coefficient.
  • characteristic features have been identified in the acquired representation, which coincide with, e.g., the onset of a syllable, the onset of a pause between syllables or words, the termination of a syllable, or the termination of a pause between syllables or words, these may be used for identifying syllables in an acquired representation of user 1 10 subvocalizing at least a part of a displayed text, such as presentation 131 displayed by computing device 100 or 200, and calculating the reading speed as the number of syllables per unit time.
  • reading continuity may be defined a measure of the fraction of syllables, or words, of presentation 131 which are actually read and subvocalized by user 1 10.
  • the reading continuity may be measured by utilizing the so-called sliding dot-product or the sliding inner-product, which is commonly used in signal processing for searching a long signal for a shorter feature.
  • the acquired representation of user 1 10 subvocalizing text is typically shorter than the expected phrasing graph, as the latter represents the entire displayed text, e.g.,
  • the subvocalized part of presentation 131 may be identified by "sliding" consecutive parts of the acquired representation of user 1 10 subvocalizing text over the time-dependent function representing the expected phrasing, by varying a shift in time between the two.
  • each of the consecutive part is of a length corresponding to a few syllables, or up to a few words.
  • the length of the consecutive parts of the acquired representation of user 1 10 subvocalizing text may be varied. The purpose is to identify the value for the time shift which maximizes the correlation between a consecutive part of the acquired representation and the time-dependent function representing the expected phrasing.
  • time-dependent function 404 illustrates an expected phrasing if user 1 10 subvocalizes "example sentences" and “read them”, whereas the other words are omitted.
  • the reading continuity may be calculated as the fraction of syllables which user 1 10 actually has subvocalized in comparison with the number of syllables comprised in presentation 131 . Based on the presented example definition for reading continuity, a measured reading continuity with a value close to unity, e.g., 0.9 or larger, indicates that user 1 10 has read and subvocalized substantially the entire
  • a considerably lower value e.g., ranging from 0.1 to 0.3, indicates that user 1 10 has omitted words, parts of sentences, or parts of text passages or paragraphs, when reading presentation 131 .
  • reading continuity may be based on the observation that people, when reading text, sometimes read consecutive passages of the text which comprise a complete sentence or more, but omit parts in-between. Accordingly, the reading continuity may be defined as the fraction of complete sentences, or substantially complete sentences, which are subvocalized in comparison with the number of sentences comprised on presentation 131 .
  • subvocal activity of user 1 10 may also be used as reading parameter.
  • Subvocal activity may be defined as the rate of subvocalization of user 1 10 during reading.
  • embodiments of the invention relying on subvocal activity as reading parameter do not require knowledge of the exact position of where in the text user 1 10 is reading. Nevertheless, subvocal activity can then still give information about user 1 10's reading manner, such as understanding and attention while reading.
  • Computing device 100 or 200 may further be operative to adapt displayed presentation 131 based on a comparison of the measured reading parameters and one or more corresponding reference reading parameters. For instance, a measured reading speed may be compared to a reference reading speed. Correspondingly, a measured reading continuity may be compared to a reference reading continuity.
  • each reference reading parameter may comprise one or more threshold values and/or one or more ranges for defining conditions which trigger adaption of
  • the one or more reference reading parameters may be associatively stored with the text, e.g., as metadata together with the text in an electronic document, i.e., file.
  • the one or more reference reading parameters may be stored in a database, either an external data base or a database comprised in computing device 100 or 200.
  • the one or more reference reading parameters may, e.g., be provided by an author or publisher of the text, and may be used for defining the intended purpose of a text, such as providing an overview over a subject, providing detailed information about a subject, or gaining the reader's interest in a subject.
  • computing device 100 or 200 may further be operative to derive the one or more reference reading parameters from acquired representations of user 1 10 subvocalizing text during a learning or training phase, similar to what is described hereinbefore. More specifically, computing device 100 or 200 may be operative to measure the one or more reading parameters from the acquired representations and store the measured reading parameters as reference reading parameters. Optionally, the thereby derived and stored one or more reference reading parameters are dependent on a type of the text. This may be achieved by displaying different types of text to user 1 10 during a learning or training phase, and measuring and storing one or more reference reading parameters in association with a text type. This is advantageous since different types of texts are typically read in a different manner. For instance, technical documents or reports are typically read with attention to detail, at rather low speed and with great continuity. News on social networks, on the other hand, are oftentimes read with less attention to detail, and a correspondingly larger reading speed and lower reading continuity, respectively.
  • a further embodiment 500 of the computing device for displaying a text to user 1 10 is illustrated.
  • Computing device 500 is similar to computing devices 100 and 200 described hereinbefore, but is operative to add information from the text to presentation 131 or remove information from presentation 131 .
  • computing device 500 is illustrated in Fig. 5A as displaying a presentation 531 of the text with a relatively high level of detail, including equations.
  • computing device 500 may adapt displayed presentation 531 by removing sentences which are characterized by a high level of detail, as well as the equations, resulting in an adapted
  • computing device 500 may be operative to first display a less detailed presentation 532 (see Fig. 5B) and adapting presentation 532 by adding information, resulting in
  • Fig. 5A based on measured reading parameters which indicate that user 1 10 is reading presentation 532 with attention to detail. For instance, this may be the case if the measured reading speed is relatively low and the measured reading continuity is relatively high, indicating that user 1 10 is reading carefully.
  • FIG. 6 a further embodiment 600 of the computing device for displaying a text to user 1 10 is illustrated.
  • Computing device 600 is similar to computing devices 100 and 200 described hereinbefore, but is operative to adapt a displayed presentation 531 by providing a summary of at least a part of the text. For instance, this may be the case if it can be concluded, based on the measured reading parameters that user 1 10 is reading a
  • presentation 531 displayed by computing device 600 hastily, i.e., with relatively high reading speed and low reading continuity, indicating that user 1 10 is omitting parts of presentation 531 when reading.
  • Computing device 600 may be operative to provide a summary 632 in addition to personation 531 , e.g., by overlaying summary 632 onto presentation 531 , as is illustrated in Fig. 6, or by replacing presentation 531 with summary 632.
  • a further embodiment 700 of the computing device for displaying a text to user 1 10 is illustrated.
  • Computing device 700 is similar to computing devices 100 and 200 described hereinbefore, but is operative to adapt a displayed presentation of the text, such as presentation 531 shown in Fig. 5A, by increasing a font size of the displayed text, such as text 732 illustrated in Fig. 7, or by magnifying the displayed text. For instance, this may be the case if the measured reading parameters reflect an unusually low reading speed, which may be an indication that user 1 10 has difficulties deciphering presentation 531 .
  • a further embodiment 800 of the computing device for displaying a text to user 1 10 is illustrated.
  • Computing device 800 is similar to computing devices 100 and 200 described hereinbefore, but is operative to adapt a displayed presentation, such as presentation 531 shown in Fig. 5A, by adding one or more references to additional information.
  • the one or more references may, e.g., be provided by hyperlinks, such as Uniform Resource Locators (URLs), indicated as underlined words or phrases in
  • An embodiment of the computing device for displaying a text to user 1 10, such as computing device 100, 200, 500, 600, 700, or 800, in the following referred to as 100-800, may further be operative to adapt the displayed presentation of the text further based on a type of the text.
  • the type of the text is associatively stored with the text, e.g., as metadata together with the text in an electronic document.
  • computing devices 100-800 are in the drawings illustrated as tablets or smartphones, embodiments of the invention may alternatively be embodied as a mobile phone, a mobile terminal, a personal computer, a laptop, an e-book reader, a computer display, a television, or a media player.
  • processing means 103 comprised in embodiments 100-800 of the computing device for displaying a text to a user of the computing device, are described with reference to Figs. 9 and 10.
  • Processing means 900 comprises a processing unit 902, such as a processing circuitry or general purpose processor, and a computer-readable storage medium 903, such as a Random Access Memory (RAM), a Flash memory, or the like.
  • processing means 900 comprises one or more interfaces 901 ("I/O" in Fig. 9) for controlling and/or receiving
  • interface(s) 901 may be operative to acquire, from one or more sensors 120 operatively connected to computing device 100-800, nerve signals captured from throat 1 1 1 of user 1 10 subvocalizing at last a part of a displayed presentation 131 of the text.
  • interface(s) 901 may be operative to acquire, from camera 102, a video sequence of user 1 10 subvocalizing at least a part of presentation 131 .
  • Memory 903 contains computer-executable
  • instructions 904 i.e., a computer program, for causing computing 100-800 device to become operative to perform in accordance with embodiments of the invention as described herein, when computer-executable
  • instructions 904 are executed on processing unit 902.
  • processing means 1000 comprises one or more interfaces 1001 ("I/O" in Fig. 10) for controlling and/or receiving information from other components comprised in the computing device, such as display 101 , camera 102, communications module 104, and interface 105 for external devices.
  • interface(s) 1001 may be operative to acquire, from one or more sensors 120 operatively connected to computing device 100-800, nerve signals captured from throat 1 1 1 of user 1 10 subvocalizing at least a part of a displayed presentation 131 of the text.
  • interface(s) 1001 may be operative to acquire, from camera 102, a video sequence of user 1 10 subvocalizing at least a part of presentation 131 .
  • Processing means 1000 further comprises a presentation module 1002, an acquisition module 1003, and a reading parameter module 1004, which are configured to cause computing device 100-800 to perform in accordance with embodiments of the invention as described herein.
  • presentation module 1002 is configured to display a presentation 131 of the text
  • acquisition module 1003 is configured to acquire a representation of user 1 10 subvocalizing at least a part of presentation 131
  • reading parameter module 1004 is configured to measure one or more reading parameters from the acquired representation, which reading parameters characterize a manner in which user 1 10 is reading
  • presentation module 1002 is further configured to adapt displayed presentation 131 based on the measured reading
  • Acquisition module 1003 may be configured to acquire a
  • representation of user 1 10 subvocalizing at least a part of presentation 131 by acquiring, from one or more sensors 120 operatively connected to computing device 100-800, nerve signals captured from a throat 1 1 1 of user 1 10 subvocalizing the part of presentation 131 , and deriving the representation of user 1 10 subvocalizing at least a part of presentation 131 text as a representation of the nerve signals.
  • acquisition module 1003 may be configured to acquire a representation of user 1 10 subvocalizing at least a part of presentation 131 by acquiring, from
  • camera 102 operatively connected to computing device 100-800, a video sequence of user 1 10 subvocalizing the part of presentation 131 , magnifying, by video processing the acquired video sequence, motions of one or more body parts 1 1 1-1 15 of user 1 10, which motions are correlated with the subvocalized part of presentation 131 , and deriving the representation of user 1 10 subvocalizing at least a part of presentation 131 as a representation of the magnified motions.
  • Reading parameter module 1004 may be configured to measure the one or more reading parameters as one or more of a reading speed, a reading continuity, and a combination thereof.
  • Presentation module 1002 may further be configured to adapt displayed presentation 131 based on a comparison of the measured reading parameters and one or more corresponding reference reading parameters.
  • the one or more reference reading parameters may be
  • Reading parameter module 1004 may further be configured to derive the one or more reference reading parameters from acquired representations of user 1 10 subvocalizing text by measuring one or more reading parameters from the acquired representations, which reading parameters characterize a manner in which user 1 10 is reading text, and storing the measured reading parameters as the reference reading parameters.
  • the one or more reference reading parameters may be dependent on a type of the text.
  • Presentation module 1002 may further be configured to adapt displayed presentation 131 further based on a type of the text.
  • Presentation module 1002 may further be configured to adapt displayed presentation 131 by adding information and/or removing
  • Presentation module 1002 may further be configured to adapt displayed presentation 131 by providing a summary of at least a part of the text.
  • Presentation module 1002 may further be configured to adapt displayed presentation 131 by increasing a font size of the displayed text and/or magnifying the displayed text.
  • Presentation module 1002 may further be configured to adapt displayed presentation 131 of the text by adding one or more references to additional information.
  • Presentation module 1002 may further be configured to adapt displayed presentation 131 of the text by changing a layout of the presented text.
  • Interface(s) 901 and 1001 , and modules 1002-1004, as well as any additional modules comprised in processing means 1000, may be
  • embodiments 1 100 of the method of displaying a text to a user 1 10 of a computing device are described with reference to Fig. 1 1 , in accordance with embodiments of the invention.
  • An embodiment of method 1 100 may be performed by a computing device such as a mobile phone, a mobile terminal, a smartphone, a tablet, a personal computer, a laptop, an e-book reader, a computer display, a television, or a media player.
  • Method 1 100 comprises displaying 1 101 a presentation 131 of the text, acquiring 1 102 a representation of user 1 10 subvocalizing at least a part of presentation 131 , measuring 1 103 one or more reading parameters from the acquired representation, which reading parameters characterize a manner in which user 1 10 is reading presentation 131 , and adapting 1 104 displayed presentation 131 based on the measured reading parameters.
  • the one or more reading parameters may be one or more of a reading speed, a reading continuity, or a combination thereof.
  • adapting 1 104 displayed presentation 131 is based on a comparison of the measured reading parameters and one or more
  • the one or more reference reading parameters may be associatively stored with the text.
  • Method 1 100 may further comprise deriving the one or more reference reading parameters from acquired representations of user 1 10 subvocalizing text by measuring one or more reading parameters from the acquired representations, which reading parameters characterize a manner in which user 1 10 is reading text, and storing the measured reading parameters as the reference reading parameters.
  • the one or more reference reading parameters may be dependent on a type of the text.
  • adapting 1 104 displayed presentation 131 may further be based on a type of the text.
  • Acquiring 1 102 a representation of user 1 10 subvocalizing at least a part of presentation 131 may comprise acquiring, from one or more sensors 120 operatively connected to the computing device, nerve signals captured from a throat 1 1 1 of user 1 10 subvocalizing the part of
  • acquiring 1 102 a representation of user 1 10 subvocalizing at least a part of presentation 131 may comprise acquiring, from a camera 102 operatively connected to the computing device, a video sequence of user 1 10 subvocalizing the part of presentation 131 , magnifying, by video processing the acquired video sequence, motions of one or more body parts 1 1 1-1 15 of user 1 10, which motions are correlated with the subvocalized part of presentation 131 , and deriving the representation of user 1 10 subvocalizing at least a part of presentation 131 as a representation of the magnified motions.
  • adapting 1 104 displayed presentation 131 may comprise adding information and/or removing information.
  • adapting 1 104 displayed presentation 131 may comprise providing a summary of at least a part of the text.
  • adapting 1 104 displayed presentation 131 may comprise increasing a font size of the displayed text and/or magnifying the displayed text.
  • adapting 1 104 displayed presentation 131 may comprise adding one or more references to additional information.
  • adapting 1 104 displayed presentation 131 may comprise changing a layout of the presented text.
  • method 1 100 may comprise additional, or modified, steps in accordance with what is described throughout this disclosure.
  • An embodiment of method 1 100 may be implemented as software, such as computer program 904, to be executed by a processing unit or processing circuitry comprised in the computing device, whereby the computing device becomes operative to perform in accordance with embodiments of the invention described herein.

Abstract

A computing device (200) for displaying a text to a user (110) is provided. The computing device is operative to display a presentation (131) of the text, acquire a representation of the user subvocalizing at least a part of the presentation of the text, measure one or more reading parameters from the acquired representation, which characterize a manner in which the user is reading the presentation of the text, and adapt the displayed presentation of the text based on the measured reading parameters. The displayed presentation may, e.g., be adapted by adding information or removing information, by providing a summary, by increasing a font size of the displayed text or magnifying the displayed text, by adding one or more references to additional information, or by changing a layout of the presented text. Thereby, the displayed text is adapted so as improve the reading experience of the user and/or to better support the purpose of the text.

Description

DISPLAYING TEXT TO A USER OF A COMPUTING DEVICE
Technical field The invention relates to a computing device for displaying a text to a user of the computing device, a method of displaying a text to a user of a computing device, a corresponding computer program, and a corresponding computer program product. Background
It is known to adapt content for display by computing devices, such as text and images, based on both the user's context and the content's usage semantics ("Context-Aware Interactive Content Adaptation", by I. Mohomed, J. C. Cai, S. Chavoshi, and E. de Lara, Proceedings of the 4th international conference on Mobile systems, applications and services, MobiSys Ό6, pages 42-55, ACM New York, 2006). This is particularly important for displaying content on mobile devices, such as smartphones and tablets, which have limitations in screen size, processing power, battery life, and wireless connectivity.
The user's context relates to properties of the computing device which is being used for displaying content, e.g., a size and/or resolution of the display, or an available bandwidth over a wireless connection, e.g., via a cellular mobile communications network or a Wireless Local Area Network (WLAN)/Wi-Fi network. The usage semantics of the content, on the other hand, relates to the task for which the content is being used, and is typically user specific ("URICA: Usage-awaRe Interactive Content Adaptation for Mobile Devices", by I. Mohomed, J. C. Cai, and E. de Lara, Proceedings of the 1 st ACM SIGOPS/EuroSys European Conference on Computer Systems 2006, EuroSys Ό6, pages 345-358, ACM New York, 2006). Hence, adaptation of content for display is dependent on the user viewing the content as well as a task the user is performing, such as scanning through a text for finding relevant information, quickly reading a text for grasping a theme of the text, or carefully reading a text for learning.
Known solutions for adapting content for display based on the content's usage semantics are based on feedback from the user, which may be learned for subsequently adapting content for display not only for the user but also other users ("URICA: Usage-awaRe Interactive Content Adaptation for Mobile Devices", by I. Mohomed, J. C. Cai, and E. de Lara, Proceedings of the 1 st ACM SIGOPS/EuroSys European Conference on Computer
Systems 2006, EuroSys Ό6, pages 345-358, ACM New York, 2006). More specifically, the user's modifications of displayed content are considered as corrective feedback on the system's adaptation decision, and these corrections are used to improve the quality of adaptation predictions for future accesses to the content by the same or other users.
It is also known to predict reading comprehension from eye-gaze data for adapting online learning content ("Predicting reading comprehension scores from eye movements using artificial neural networks and fuzzy output error", by L. Copeland, T. Gedeon, and S. Mendis, Artificial Intelligence Research, vol. 3, pages 35^18, Sciedu Press, 2014). Eye-tracking is also used for document summarization algorithms which rely on the reading time of individual users spent on single words in a document as the essential clue ("User-Oriented Document Summarization through Vision-Based Eye- Tracking", by S. Xu, H. Jiang, and F. C. M. Lau, Proceedings of the 14th international conference on Intelligent user interfaces, IUI Ό9, pages 7-16, ACM New York, 2009). Summary
It is an object of the invention to provide an improved alternative to the above techniques and prior art.
More specifically, it is an object of the invention to provide an improved adaption of text for display to a user of a computing device. In particular, it is an object of the invention to provide an improved adaption which is based on the manner the user is reading a displayed text.
These and other objects of the invention are achieved by means of different aspects of the invention, as defined by the independent claims. Embodiments of the invention are characterized by the dependent claims.
According to a first aspect of the invention, a computing device for displaying a text to a user of the computing device is provided. The computing device comprises processing means which is operative to display a presentation of the text, acquire a representation of the user subvocalizing at least a part of the presentation of the text, and measure one or more reading parameters from the acquired representation. The reading
parameters characterize a manner in which the user is reading the
presentation of the text. The processing means is further operative to adapt the displayed presentation of the text based on the measured reading parameters.
According to a second aspect of the invention, a method of displaying a text to a user of a computing device is provided. The method is performed by the computing device and comprises displaying a presentation of the text, acquiring a representation of the user subvocalizing at least a part of the presentation of the text, and measuring one or more reading parameters from the acquired representation. The reading parameters characterize a manner in which the user is reading the presentation of the text. The method further comprises adapting the displayed presentation of the text based on the measured reading parameters. According to a third aspect of the invention, a computer program is provided. The computer program comprises computer-executable
instructions for causing a device to perform the method according to an embodiment of the second aspect of the invention, when the computer- executable instructions are executed on a processing unit comprised in the device.
According to a fourth aspect of the invention, a computer program product is provided. The computer program product comprises a computer- readable storage medium which has the computer program according to the third aspect of the invention embodied therein.
The invention makes use of an understanding that an improved adaption of content, in particular text, to a user of a computing device may be achieved based on subvocalization.
Subvocalization is the "internal speech" while reading a text and is characterized by minute movements of the larynx and other speech muscles, as well as other body parts, which are imperceptible to the human eye but detectable by technical means. By acquiring a representation of the user subvocalizing at least a part of a displayed presentation of a text he or she is reading, and measuring one or more reading parameters from the acquired representation, which reading parameters characterize a manner in which the user is reading the presentation of the text, the displayed presentation of the text may be adapted based on the measured reading parameters. In the present context, a presentation of a text is to be understood as to encompass the information contained in the text, or a subset thereof, as well as the way it is presented to the user, i.e., displayed. For instance, with respect to the information contained in a text, different presentations of a text may, e.g., be the complete text, one or more consecutive parts of the text, e.g., sentences, or a summary of the text. With respect to the way a text is presented to the user, different presentations of the text may, e.g., utilize different layouts, fonts, styles, and magnification. The one or more reading parameters may, e.g., be a reading speed, a reading continuity, a subvocal activity, or a combination thereof. In addition, one of the reading parameters may also reflect whether the user is reading the text in the order it is written, or "hopping" back and forth when reading. The latter may, e.g., indicate that the user is searching for information.
Embodiments of the invention are advantageous since people increasingly prefer viewing content, such as text and images, which are displayed by a computing device, either on a built-in screen or an external screen, rather than a printed paper copy. The computing device may, e.g., be a mobile phone, a mobile terminal, a smartphone, a tablet, a laptop, an e- book reader, a television, or a media player, which typically are provided with built-in screens, oftentimes of touchscreen type. Alternatively, the computing device may be a personal computer which is operatively connected to an external a computer display.
Advantageously, embodiments of the invention enable an improved adaption of text displayed to a user based on measured reading parameters which reflect the manner in which the user is reading the text. The user's manner in reading the text is related to a task the user is performing, such as scanning through a text for finding relevant information, quickly reading through a text for grasping a theme gist of the text, or carefully reading through a text for learning, but may also be related to the user's attitude towards the text, i.e., the user's comprehension of the text, his/her interest in the text of the information contained therein, and the user's focus when reading a presentation of the text. The presentation of the text is adapted so as improve the reading experience of the user and/or to better support the purpose of the text, such as providing an overview over a subject, providing detailed information about a subject, or gaining the user's interest in a subject.
According to an embodiment of the invention, the representation of the user subvocalizing at least a part of the presentation of the text may be acquired by acquiring nerve signals which are captured from a throat of the user subvocalizing the part of the presentation of the text, and deriving the representation of the user subvocalizing at least a part of the presentation of the text as a representation of the nerve signals. The nerve signals are acquired from one or more sensors, or electrodes, which are operatively connected to the computing device, in particular electromyographical (EMG) sensors or electrodes. As a person reads a text, even without vocalizing it audibly, they still make a variety of internal and external motions which are triggered by nerve signals controlling the muscles of the person. Both the nerve signals and the resulting motions, which typically are imperceptible to the human eye, are correlated with the subvocalized text and can be detected. In particular, a single subvocal movement has strong correlations to a single syllable, and the number of syllables per unit of time, which is a measure of reading speed, can be detected by measuring the speed of subsequent subvocal movements. Moreover, when the subvocal movements follow each other with a speed above a certain threshold, this indicates that they belong to the same word. When the detected subvocal activity is below a certain threshold, it can be concluded that the user makes a pause since he or she has reached a punctuation mark in the text, or because he/she is not reading. These internal and external motions include muscle movements which are related to the ones the person would make to audibly create the sounds, such as the movement of vocal chords, and may also include associated movements the person would make, e.g., sway due to a rhythm of the text and modified breathing patterns. Examples of these motions are motions of vocal chords, lips, tongue, jaw, neck, other body parts such as heads and shoulders, and motions related to breathing.
According to another embodiment of the invention, the representation of the user subvocalizing at least a part of the presentation of the text may be acquired by acquiring a video sequence of the user subvocalizing the part of the presentation of the text, magnifying motions of one or more body parts of the user, and deriving the representation of the user subvocalizing at least a part of the presentation of the text as a representation of the magnified motions. The motions of the one or more body parts are correlated with the subvocalized part of the presentation of the text. The representation of the magnified motions may, e.g., be derived using one or more metrics
quantifying the motions of the lips, throat, or other body parts, of the user. The video sequence is acquired from a camera operatively connected to the computing device, e.g., a camera which is comprised in the computing device, such as a front-facing camera in a smartphone or tablet. Alternatively, an external camera, such as a web cam which is connected to the computing device via an interface may be used. The camera captures motions of the one or more body parts of the user, such as the user's lips, throat, mouth, nose, face, neck, or shoulders. The motions may, e.g., be magnified by means of Eulerian Video Magnification (EVM) or Lagrangian techniques which are described in further detail below. Acquiring the representation of at least a part of the presentation of the text subvocalized by the user by means of a video capturing the user subvocalizing when reading, and processing the video to magnify motions which are correlated with the subvocalized text, is an alternative to utilizing EMG sensors for capturing nerve signals.
According to an embodiment of the invention, the displayed
presentation of the text is adapted based on a comparison of the measured reading parameters and one or more corresponding reference reading parameters. For instance, if reading speed if measured as one of the reading parameters, the measured reading speed may be compared to a
corresponding reference reading speed. Correspondingly, if reading continuity is measured as another of the reading parameters, indicating the fraction of displayed syllables or words which are actually read and
subvocalized by the user, the measured reading continuity may be compared to a corresponding reference reading continuity. Typically, each reference reading parameter may comprise one or more threshold values and/or one or more ranges for defining conditions which trigger adaption of a displayed presentation of a text based on the subvocalization of the user when reading the displayed presentation. As an example, if he measured reading speed is above a certain reading-speed threshold value and the measured reading continuity is below a certain reading-continuity threshold value, it may be concluded that the user is reading hastily through the displayed presentation of the text and is likely to prefer a less detailed presentation of the text.
According to an embodiment of the invention, the one or more reference reading parameters may be associatively stored with the text. For instance, the one or more reference reading parameters may be stored as meta data, together with the text in an electronic document, or in a separate database. Thereby, the author or publisher of a text can define how the text is to be read.
According to an embodiment of the invention, the one or more reference reading parameters may be derived from acquired representations of the user subvocalizing text by measuring one or more reading parameters from the acquired representations and storing the measured reading parameters as the reference reading parameters. In other words, the reference reading parameters are derived in a similar manner as the reading parameters based on which the displayed presentation of the text is adapted. Preferably, the one or more reference reading parameters are derived from one or more texts which are displayed to, and read by, the user during a training or learning phase, and may optionally be derived during usage of the computing device. As an alternative, the one or more reference reading parameters may be configured by the user, a manufacturer of the device, or provided together with the text. Optionally, the one or more reference reading parameters may be dependent on a type of the text. This is advantageous since different types of texts are typically read in a different manner. For instance, technical documents or reports are typically read with attention to detail, at rather low speed and with great continuity. News on social networks, on the other hand, are oftentimes read with less attention to detail, and a correspondingly larger reading speed and lower reading continuity, respectively.
According to an embodiment of the invention, the displayed
presentation of the text is adapted further based on a type of the text. This is advantageous since different types of texts have different reading styles, i.e., the manner in which a user is reading a text, associated with it. For instance, a report may be read with attention to detail by one user, characterized by relatively low reading speed and relatively high reading continuity, whereas it may be read with less attention to detail by a different user which is interested in the gist of the text.
According to an embodiment of the invention, the displayed
presentation of the text may be adapted by adding information from the text to the presentation of the text, or by removing information from the displayed presentation of the text. Moreover, information from the text may be added in one part of the presentation and information may be removed in another part. In other words, the level of detail which is presented to the user may be adjusted based on the measured reading parameters. For instance, if the manner in which the user is reading and subvocalizing the displayed presentation of the text is characterized by a relatively low reading speed and a relatively high reading continuity, it may be concluded that the user reads the presented text with attention to detail, and the displayed presentation of the text may be adapted by adding additional information, e.g., additional passages providing a higher level of detail. On the other hand, if the manner in which the user is reading and subvocalizing the displayed presentation of the text is characterized by a relatively high reading speed and a relatively low reading continuity, it may be concluded that the user reads the presented text with little attention to detail and is likely attempting to extract the gist of the text. In this case, the displayed presentation of the text may be adapted by removing information which the user most likely is not interested in, e.g., passages providing a higher level of detail.
According to an embodiment of the invention, the displayed
presentation of the text may be adapted by providing a summary of at least a part of the text. This is advantageous if the manner in which the user is reading and subvocalizing the presentation of the text is characterized by a relatively high reading speed and a relatively low reading continuity, indicating that the user reads the presented text with little attention to detail and is likely attempting to extract the gist of the text. In this case, the displayed presentation of the text may be amended, or replaced, by a summary of the text. The summary may either be provided with the text, e.g., as metadata or as a separate section in an electronic document, such as a file, together with the text, or generated using Natural Language Processing (NLP) techniques.
According to an embodiment of the invention, the displayed
presentation of the text may be adapted by increasing a font size of the displayed text and/or magnifying the displayed text. This may be
advantageous in situations if the measured reading parameters indicate an unusually low reading speed, which may be an indication that the user has difficulties deciphering the displayed presentation of the text.
According to an embodiment of the invention, the displayed
presentation of the text may be adapted by adding one or more references to additional information, such as hyperlinks or tool tips. For instance, if the measured reading parameters indicate that the user is reading some parts of the presentation of the text with attention to detail, characterized by a relatively low reading speed and a relatively high reading continuity, whereas other parts are read with less attention to detail, the displayed presentation may be adapted by reducing the level of detail, i.e., providing less information in the displayed presentation but adding hyperlinks or tooltips for giving the user the opportunity to access additional information. According to an embodiment of the invention, the displayed presentation of the text may be adapted by changing a layout of the presented text. This is advantageous in situations when the measured reading parameters indicate that the user is hopping back and forth in the displayed presentation, attempting to find specific information. By presenting the parts of the text in different order, the reading experience of the user may be improved.
Even though advantages of the invention have in some cases been described with reference to embodiments of the first aspect of the invention, corresponding reasoning applies to embodiments of other aspects of the invention.
Further objectives of, features of, and advantages with, the invention will become apparent when studying the following detailed disclosure, the drawings, and the appended claims. Those skilled in the art realize that different features of the invention can be combined to create embodiments other than those described in the following.
Brief description of the drawings The above, as well as additional objects, features and advantages of the invention, will be better understood through the following illustrative and non-limiting detailed description of embodiments of the invention, with reference to the appended drawings, in which:
Fig. 1 shows a computing device for displaying a text to a user of the computing device, in accordance with an embodiment of the invention.
Fig. 2 shows a computing device for displaying a text to a user of the computing device, in accordance with another embodiment of the invention.
Fig. 3 exemplifies an image of a video sequence captured by a camera comprised in the embodiment of the computing device shown in Fig. 2. Fig. 4 illustrates an example text, an expected phrasing when read by a user, and a representation of the expected phrasing as a time-dependent function.
Figs. 5A and 5B illustrate adapting a displayed presentation of a text by adding or removing information, in accordance with an embodiment of the invention.
Fig. 6 illustrates adapting a displayed presentation of a text by providing a summary of at least a part of the text, in accordance with another embodiment of the invention.
Fig. 7 illustrates adapting a displayed presentation of a text by increasing a font size of the displayed text, in accordance with a further embodiment of the invention.
Fig. 8 illustrates adapting a displayed presentation of a text by adding one or more references to additional information, in accordance with yet another embodiment of the invention.
Fig. 9 shows an embodiment of the processing means comprised in the computing device for displaying a text to a user.
Fig. 10 shows another embodiment of the processing means comprised in the computing device for displaying a text to a user.
Fig. 1 1 shows a method of displaying a text to a user of the computing device, in accordance with embodiments of the invention.
All the figures are schematic, not necessarily to scale, and generally only show parts which are necessary in order to elucidate the invention, wherein other parts may be omitted or merely suggested.
Detailed description
The invention will now be described more fully herein after with reference to the accompanying drawings, in which certain embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
The embodiments of the invention described herein make use of an understanding that text which is displayed to a user of a computing device may be adapted based on detecting, using an acquired representation of a subvocalization by the user, to what extent and/or how the user actually is reading displayed text. This is achieved by measuring one or more reading parameters which characterize a manner in which the user is reading the presentation of the text. By adapting the displayed presentation of the text based on the one or more measured reading parameters, the displayed presentation of the text can advantageously be improved so as to better fit the reading style of the user, the task at hand, and/or the intended purpose of the text. For instance, the user may be scanning through a text for finding relevant information, quickly reading a text for grasping a theme of the text, or carefully reading a text for learning. The intended purpose of a text may, e.g., be to provide an overview over a subject, to provide detailed information about a subject, or to gain the user's interest in a subject.
Subvocalization is the "internal speech" while reading a text and is characterized by minute movements of the larynx and other speech muscles, as well as other body parts, which are imperceptible to the human eye but detectable by technical means. As a person reads a text, although they do not make audible noise, they still make a variety of internal (e.g., vocal chords) and external (e.g., lips) motions which can be detected, albeit being imperceptible to the human eye. These motions include muscle movements which are related to the ones the person typically would make to create audible sounds, and which consequently are correlated with the text subvocalized by the user. To give some examples, the movements include: - Lip motions: Each time a person says a syllable their lips move. The pronunciation of each syllable requires the user to move their lips. For instance, pronouncing the word "sen-tence" requires to first pronounce "sen" and then again to pronounce "tence". - Throat movement: The utterance of a syllable also requires the user to change the shape of their vocal chords. This is done by moving the muscles that control the vocal chords, and this movement in turn moves the surface of the user's throat.
- Pauses: A person who is reading pauses in between words, and also at punctuation points such as comma, question mark, and full stop. The person may also breath during a pause, resulting in modified breathing patterns.
- Uncertainty: If a person is pronouncing a word they are less certain about it may take longer as they need cognitive effort to work out what the word is.
In addition, a person who is reading may also sway to the text he/she is reading, in particular if the text is lyrics of a piece of music or poetry.
In Fig. 1 , an embodiment 100 of the computing device for displaying a text to a user 1 10 of computing device 100 is shown. Computing device 100 is operative to adapt a displayed presentation 131 of the text based on one or more reading parameters which are measured from an acquired
representation of user 1 10 subvocalizing at least a part of presentation 131 . The one or more reading parameters characterize a manner in which user 1 10 is reading presentation 131 and may, e.g., be a reading speed, a reading continuity, or a combination thereof. Computing device 100 is in
Fig. 1 illustrated as a tablet comprising a display 101 , e.g., a touchscreen, a front-facing camera 102, processing means 103, a communications module 104, and an optional interface 105 for connecting external devices, as is described further below. Communications module 104 is operative to effect wireless
communications with a Radio Access Network (RAN) or with another computing device, based on a cellular telecommunications technique such as the Global System for Mobile communications (GSM), Universal Mobile Telecommunications System (UMTS), Long Term Evolution (LTE), or any 5G standard. Alternatively, or additionally, communications module 104 may be operative to effect wireless communications through a Wireless Local Arena Network (WLAN)/Wi-Fi network.
Interface 105 for connecting external devices is in Fig. 1 illustrated as a conventional headphone jack, or headset jack, but may alternatively be embodiment as any other type of connector, such as a Lightning connector, a Universal Serial Bus (USB) connector, or the like. As yet a further alternative, interface 105 may also be a wireless interface based on technologies like coded visible or invisible light, Bluetooth, or ZigBee.
Computing device 100 is operative to display presentation 131 of the text to user 1 10, either using built-in display 101 or an external display which is operatively connected to computing device 100, e.g., a computer display. Computing device 100 is further operative to acquire a representation of user 1 10 subvocalizing at least a part of presentation 131 , and to measure one or more reading parameters from the acquired representation. The measured one or more reading parameters characterize a manner in which user 1 10 is reading presentation 131 . Computing device 100 is further operative to adapt presentation 131 text based on the measured reading parameters.
More specifically, and with reference to Fig. 1 , computing device 100 is operative to acquire the representation of user 1 10 subvocalizing at least a part of presentation 131 by acquiring nerve signals which are captured from the throat 1 1 1 of user 1 10 when subvocalizing at least a part of
presentation 131 . The nerve signals may be acquired by means of one or more EMG sensors 120, or electrodes, which are attached to throat 1 1 1 , e.g., one sensor 120 on each side of the larynx, and which are operatively connected to computing device 100, e.g., via interface 105. The captured nerve signals are correlated with the part of presentation 131 which user 1 10 subvocalizes when reading, since they control movements of the larynx and other body parts 1 1 1-1 15 of user 1 10. The nerve signals which are acquired from sensors 120 may either be received directly, e.g., as an analog electric signal, as a digitized representation of the analog sensor signal, or as a chunk of data captured by sensors 120 and delivered to computing device 100, optionally after pre-processing, either wired or wirelessly via interface 105. The representation of user 1 10 subvocalizing at least a part of presentation 131 is derived as a representation of the nerve signals, and can subsequently be used for measuring the one or more reading parameters, as is described further below with reference to Fig. 4.
In Fig. 2, another embodiment 200 of the computing device for displaying a text to a user 1 10 of computing device 100 is shown. Similar to computing device 100 described with reference to Fig. 1 , computing device 200 comprises a display 101 , e.g., a touchscreen, a front-facing camera 102, processing means 103, a communications module 104, and an optional interface 105 for connecting external devices.
In correspondence with what is described with reference to Fig. 1 , computing device 200 is operative to display presentation 131 of the text to user 1 10, either using built-in display 101 or an external display which is operatively connected to computing device 200, e.g., a computer display. Computing device 200 is further operative to acquire a representation of user 1 10 subvocalizing at least a part of presentation 131 , and to measure one or more reading parameters from the acquired representation. The measured one or more reading parameters characterize a manner in which user 1 10 is reading presentation 131 . Computing device 200 is further operative to adapt presentation 131 text based on the measured reading parameters. In contrast to what is described with reference to Fig. 1 , computing device 200 is operative to acquire the representation of user 1 10
subvocalizing at least a part of presentation 131 by acquiring a video sequence of user 1 10 subvocalizing at least a part of presentation 131 , and magnifying motions of one or more body parts 1 1 1-1 15 of user 1 10 which are correlated with the subvocalized part of presentation 131 . The video sequence is acquired from a camera operatively connected to computing device 200, such as front-facing camera 102 or an external camera
operatively connected to computing device 200, e.g., a web cam or a camera mounted in a computer display. The camera has a field-of-view 130 so as to capture at least one of the user's 1 10 face, throat 1 1 1 , mouth or lips 1 12, nose 1 13, neck 1 14, and shoulders 1 15. An example of an image of a video sequence captured by camera 102 is illustrated in Fig. 3.
More specifically, computing device 200 is operative to magnify the motions of one or more body parts 1 1 1-1 15 by video processing the acquired video sequence, either using EVM or by utilizing Lagrangian techniques. EVM is an image processing technique which applies a spatial decomposition followed by a temporal filter to the frames of a standard video in order to amplify very small motions which are present but typically are imperceptible to the human eye. The technique can be applied in real-time to highlight events occurring at specified temporal frequencies. For a
description of EVM, see, e.g., "Eulerian Video Magnification for Revealing Subtle Changes in the World", by H.-Y. Wu, M. Rubinstein, E. Shih,
J. Guttag, F. Durand, and W. Freeman (Proceedings of ACM SIGGRAPH 2012, ACM Transactions on Graphics, vol. 31 , article no. 65, ACM New York, 2012). A comparison between EVM and the Lagrangian approach has been reported in "Phase-based video motion processing", by N. Wadhwa, M.
Rubinstein, F. Durand, and W. T. Freeman (SIGGRAPH 2013 Conference Proceedings, ACM Transactions on Graphics, vol. 32, article no. 80, ACM New York, 2013), and it was concluded that the Eulerian approach, i.e., EVM, supports larger amplification factors at better noise performance. Whereas the Lagrangian approaches estimate the motions explicitly (so the motions themselves are amplified), EVM relies on calculating and amplifying the non- motion compensated frame differences. A solution for coping with large motions in the Lagrangian approach is reported in "Video magnification in presence of large motions" by M. A. Elgharib, M. Hefeeda, F. Durand, and W. T. Freeman (2015 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pages 41 19-4127, IEEE Computer Society, 2015), and is based on defining a region-of-interest within which motions are magnified.
Further with reference to Figs. 2 and 3, computing device 200 is operative to derive the representation of user 1 10 subvocalizing at least a part of presentation 131 as a representation of the magnified motions. For instance, the representation may be based on one or more metrics which quantify the motions of lips 1 12, neck 1 14, or other body parts, of user 1 10. In practice, if lips 1 12 are captured in the video sequence obtained from camera 102, the representation may be based on a measured distance 301 , magnified by video processing, between the upper and lower lip 1 12 over time. As an alternative, the representation may be based on a measured width 302, magnified by video processing, of neck 1 14 over time. The representation of user 1 10 subvocalizing the part of first text 131 can subsequently be used for measuring the one or more reading parameters, as is described in the following.
In Fig. 4, a text 401 is illustrated as an example for presentation 131 . When read by user 1 10, text 401 can be represented as a phrasing
graph 402 which graphically illustrates the phrasing which user 1 10 applies when reading text 401 . In particular, phrasing graph 402 illustrates the duration of each syllable (indicated by "_"), the separation between syllables (indicated by 7"), which is an indication for user 1 10's reading cadence, as well as the duration of pauses (indicated by "Π") at the commas and the question mark, respectively. Note that whereas phrasing graph 402 is a graphical representation of user 1 10's phrasing when reading text 401 , it will be appreciated that user 1 10's phrasing may alternatively be cast into a mathematical representation which is suitable for a more quantitative comparison with the acquired representation of user 1 10 subvocalizing at least a part of presentation 131 . For instance, the phrasing for text 401 may be represented by a time-dependent function 403, or a signal, which assumes a value of one ("1 ") during utterance of a syllable by user 1 10, and a value of zero ("0") during a pause between syllables, a pause between words, or a pause due to a punctuation mark. It will be appreciated that numerous mathematical representations of the phrasing graph for text 401 may be envisaged, and the embodiments of the invention are not limited to the particular form 403 illustrated in Fig. 4.
Computing device 100 or 200 is operative to measure the one or more reading parameters from the acquired representation of user 1 10
subvocalizing a part of presentation 131 . For instance, a reading speed, which may, e.g., be defined as the number of subvocalized syllables per unit time, may be measured by identifying syllables and pauses in the acquired representation and counting the number of identified syllables during a predetermined time interval, typically a few seconds. Syllables and pauses may be identified by comparing the acquired representation of user 1 10 subvocalizing text with an expected phrasing for the displayed text. The expected phrasing may be derived based on information which is acquired during a training or learning phase. More specifically, a text is displayed to user 1 10, and a representation of user 1 10 subvocalizing the displayed text is acquired. Then, by comparing the acquired representation to a mathematical presentation of an expected phrasing graph of the displayed text, similar to time-dependent function 403, characteristic features may be identified in the acquired representation which coincide with, e.g., the onset of a syllable, the onset of a pause between syllables or words, the termination of a syllable, the termination of a pause between syllables or words, and so forth. The comparison between the acquired representation and the expected phrasing may be performed by deriving a time-dependent function, or signal, for the displayed text, such as function 403 for text 401 , by utilizing predetermined values for the duration of syllables, pauses between syllables, pauses between words, and pauses in relation to punctuation marks, respectively. These predetermined values may, e.g., be configured by a manufacturer of computing device 100 or 200, and subsequently be improved during the training or learning phase by varying the values and calculating a correlation between the thereby obtained time-dependent function, which represents an expected phrasing for the displayed text, and the acquired representation of user 1 10 subvocalizing the displayed text. The respective values for the different durations are then varied until the calculated correlation assumes a maximum, indicating that the corresponding values for the duration of syllables, pauses between syllables, pauses between words, and pauses in relation to punctuation marks, respectively, are the best representation of user 1 10's manner of reading the displayed text, reflecting user-specific properties such as user 1 10's cadence and/or reading speed.
As is known in the art, correlation is a statistical relationship which reflects the extent to which two random variables, e.g., two time-dependent functions or signals such as a time-dependent function representing an expected phrasing by user 1 10 when subvocalizing a displayed text and the acquired representation of user 1 10 subvocalizing the displayed text, overlap. The correlation between two random variables is commonly referred to as cross-correlation and can be quantified by means of a correlation function, which can be expressed as an integral over the two random variables over time. Typically, correlation functions are normalized such that a perfect correlation between the two random variables, i.e., the two random variables are identical, result in a maximum value which oftentimes is chosen to be equal to one ("1 "). Correspondingly, the correlation of two completely independent random variables yields a correlation value of zero ("0"). An example is the well-known Pearson product-moment correlation coefficient.
After characteristic features have been identified in the acquired representation, which coincide with, e.g., the onset of a syllable, the onset of a pause between syllables or words, the termination of a syllable, or the termination of a pause between syllables or words, these may be used for identifying syllables in an acquired representation of user 1 10 subvocalizing at least a part of a displayed text, such as presentation 131 displayed by computing device 100 or 200, and calculating the reading speed as the number of syllables per unit time.
As an alternative, or in addition, to reading speed, other reading parameters which characterize a manner in which user 1 10 is reading presentation 131 may also be used in adapting presentation 131 . For instance, reading continuity may be defined a measure of the fraction of syllables, or words, of presentation 131 which are actually read and subvocalized by user 1 10. The reading continuity may be measured by utilizing the so-called sliding dot-product or the sliding inner-product, which is commonly used in signal processing for searching a long signal for a shorter feature. For embodiments of the invention, the acquired representation of user 1 10 subvocalizing text is typically shorter than the expected phrasing graph, as the latter represents the entire displayed text, e.g.,
presentation 131 , whereas the user may only have subvocalized a part of it. The subvocalized part of presentation 131 may be identified by "sliding" consecutive parts of the acquired representation of user 1 10 subvocalizing text over the time-dependent function representing the expected phrasing, by varying a shift in time between the two. Typically, each of the consecutive part is of a length corresponding to a few syllables, or up to a few words. In addition, the length of the consecutive parts of the acquired representation of user 1 10 subvocalizing text may be varied. The purpose is to identify the value for the time shift which maximizes the correlation between a consecutive part of the acquired representation and the time-dependent function representing the expected phrasing. Based on the obtained value, for each consecutive part, the location of the subvocalized part within presentation 131 may be identified. This is exemplified in Fig. 4, in which time-dependent function 404 illustrates an expected phrasing if user 1 10 subvocalizes "example sentences" and "read them", whereas the other words are omitted.
Subsequently, information about user 1 10's reading continuity may be derived. For instance, the reading continuity may be calculated as the fraction of syllables which user 1 10 actually has subvocalized in comparison with the number of syllables comprised in presentation 131 . Based on the presented example definition for reading continuity, a measured reading continuity with a value close to unity, e.g., 0.9 or larger, indicates that user 1 10 has read and subvocalized substantially the entire
presentation 131 , whereas a considerably lower value, e.g., ranging from 0.1 to 0.3, indicates that user 1 10 has omitted words, parts of sentences, or parts of text passages or paragraphs, when reading presentation 131 .
An alternative definition of reading continuity may be based on the observation that people, when reading text, sometimes read consecutive passages of the text which comprise a complete sentence or more, but omit parts in-between. Accordingly, the reading continuity may be defined as the fraction of complete sentences, or substantially complete sentences, which are subvocalized in comparison with the number of sentences comprised on presentation 131 .
As yet a further example, the subvocal activity of user 1 10 may also be used as reading parameter. Subvocal activity may be defined as the rate of subvocalization of user 1 10 during reading. Advantageously, embodiments of the invention relying on subvocal activity as reading parameter do not require knowledge of the exact position of where in the text user 1 10 is reading. Nevertheless, subvocal activity can then still give information about user 1 10's reading manner, such as understanding and attention while reading.
Computing device 100 or 200 may further be operative to adapt displayed presentation 131 based on a comparison of the measured reading parameters and one or more corresponding reference reading parameters. For instance, a measured reading speed may be compared to a reference reading speed. Correspondingly, a measured reading continuity may be compared to a reference reading continuity. Typically, each reference reading parameter may comprise one or more threshold values and/or one or more ranges for defining conditions which trigger adaption of
presentation 131 based on the subvocalization of user 1 10 when reading presentation 131 . As an example, if he measured reading speed is above a certain reading-speed threshold value and the measured reading continuity is below a certain reading-continuity threshold value, it may be concluded that user 1 10 is reading hastily through presentation 131 and is likely to prefer a less detailed presentation of the text. The one or more reference reading parameters may be associatively stored with the text, e.g., as metadata together with the text in an electronic document, i.e., file. Alternatively, the one or more reference reading parameters may be stored in a database, either an external data base or a database comprised in computing device 100 or 200. The one or more reference reading parameters may, e.g., be provided by an author or publisher of the text, and may be used for defining the intended purpose of a text, such as providing an overview over a subject, providing detailed information about a subject, or gaining the reader's interest in a subject.
As an alternative to associatively storing the reference reading parameters with the text, computing device 100 or 200 may further be operative to derive the one or more reference reading parameters from acquired representations of user 1 10 subvocalizing text during a learning or training phase, similar to what is described hereinbefore. More specifically, computing device 100 or 200 may be operative to measure the one or more reading parameters from the acquired representations and store the measured reading parameters as reference reading parameters. Optionally, the thereby derived and stored one or more reference reading parameters are dependent on a type of the text. This may be achieved by displaying different types of text to user 1 10 during a learning or training phase, and measuring and storing one or more reference reading parameters in association with a text type. This is advantageous since different types of texts are typically read in a different manner. For instance, technical documents or reports are typically read with attention to detail, at rather low speed and with great continuity. News on social networks, on the other hand, are oftentimes read with less attention to detail, and a correspondingly larger reading speed and lower reading continuity, respectively.
In the following, different ways of adapting a displayed presentation of a text, such as presentation 131 , are described with reference to Figs. 5 to 8, in accordance with embodiments of the invention.
In Figs. 5A and 5B, a further embodiment 500 of the computing device for displaying a text to user 1 10 is illustrated. Computing device 500 is similar to computing devices 100 and 200 described hereinbefore, but is operative to add information from the text to presentation 131 or remove information from presentation 131 . For instance, computing device 500 is illustrated in Fig. 5A as displaying a presentation 531 of the text with a relatively high level of detail, including equations. In response to measuring one or more reading parameters from acquired representations of user 1 10 subvocalizing at least a part of presentation 531 , computing device 500 may adapt displayed presentation 531 by removing sentences which are characterized by a high level of detail, as well as the equations, resulting in an adapted
presentation 532 (see Fig. 5B). For instance, this may be the case if it can be concluded, based on the measured reading parameters that user 1 10 is reading first presentation 531 hastily, i.e., with relatively high reading speed and low reading continuity, indicating that user 1 10 is omitting parts of presentation 531 when reading. Likewise, computing device 500 may be operative to first display a less detailed presentation 532 (see Fig. 5B) and adapting presentation 532 by adding information, resulting in
presentation 531 (see Fig. 5A), based on measured reading parameters which indicate that user 1 10 is reading presentation 532 with attention to detail. For instance, this may be the case if the measured reading speed is relatively low and the measured reading continuity is relatively high, indicating that user 1 10 is reading carefully.
In Fig. 6, a further embodiment 600 of the computing device for displaying a text to user 1 10 is illustrated. Computing device 600 is similar to computing devices 100 and 200 described hereinbefore, but is operative to adapt a displayed presentation 531 by providing a summary of at least a part of the text. For instance, this may be the case if it can be concluded, based on the measured reading parameters that user 1 10 is reading a
presentation 531 displayed by computing device 600 hastily, i.e., with relatively high reading speed and low reading continuity, indicating that user 1 10 is omitting parts of presentation 531 when reading. Computing device 600 may be operative to provide a summary 632 in addition to personation 531 , e.g., by overlaying summary 632 onto presentation 531 , as is illustrated in Fig. 6, or by replacing presentation 531 with summary 632.
In Fig. 7, a further embodiment 700 of the computing device for displaying a text to user 1 10 is illustrated. Computing device 700 is similar to computing devices 100 and 200 described hereinbefore, but is operative to adapt a displayed presentation of the text, such as presentation 531 shown in Fig. 5A, by increasing a font size of the displayed text, such as text 732 illustrated in Fig. 7, or by magnifying the displayed text. For instance, this may be the case if the measured reading parameters reflect an unusually low reading speed, which may be an indication that user 1 10 has difficulties deciphering presentation 531 . In Fig. 8, a further embodiment 800 of the computing device for displaying a text to user 1 10 is illustrated. Computing device 800 is similar to computing devices 100 and 200 described hereinbefore, but is operative to adapt a displayed presentation, such as presentation 531 shown in Fig. 5A, by adding one or more references to additional information. The one or more references may, e.g., be provided by hyperlinks, such as Uniform Resource Locators (URLs), indicated as underlined words or phrases in
presentation 832 shown in Fig. 8, or the like. For instance, this may be the case if the measured reading parameters indicate that user 1 10 is reading presentation 531 carefully, with attention to detail. Advantageously, user 1 10 may use the provided references to access further information which may provide a higher level of detail.
An embodiment of the computing device for displaying a text to user 1 10, such as computing device 100, 200, 500, 600, 700, or 800, in the following referred to as 100-800, may further be operative to adapt the displayed presentation of the text further based on a type of the text.
Preferably, the type of the text is associatively stored with the text, e.g., as metadata together with the text in an electronic document.
Whereas computing devices 100-800 are in the drawings illustrated as tablets or smartphones, embodiments of the invention may alternatively be embodied as a mobile phone, a mobile terminal, a personal computer, a laptop, an e-book reader, a computer display, a television, or a media player.
In the following, embodiments of processing means 103, comprised in embodiments 100-800 of the computing device for displaying a text to a user of the computing device, are described with reference to Figs. 9 and 10.
In Fig. 9, a first embodiment 900 of processing means 103 is shown. Processing means 900 comprises a processing unit 902, such as a processing circuitry or general purpose processor, and a computer-readable storage medium 903, such as a Random Access Memory (RAM), a Flash memory, or the like. In addition, processing means 900 comprises one or more interfaces 901 ("I/O" in Fig. 9) for controlling and/or receiving
information from other components comprised in computing device 100-800, such as display 101 , camera 102, communications module 104, and interface 105 for external devices. In particular, interface(s) 901 may be operative to acquire, from one or more sensors 120 operatively connected to computing device 100-800, nerve signals captured from throat 1 1 1 of user 1 10 subvocalizing at last a part of a displayed presentation 131 of the text. Alternatively, interface(s) 901 may be operative to acquire, from camera 102, a video sequence of user 1 10 subvocalizing at least a part of presentation 131 . Memory 903 contains computer-executable
instructions 904, i.e., a computer program, for causing computing 100-800 device to become operative to perform in accordance with embodiments of the invention as described herein, when computer-executable
instructions 904 are executed on processing unit 902.
In Fig. 10, an alternative embodiment 1000 of processing means 103 is illustrated. Similar to processing means 900, processing means 1000 comprises one or more interfaces 1001 ("I/O" in Fig. 10) for controlling and/or receiving information from other components comprised in the computing device, such as display 101 , camera 102, communications module 104, and interface 105 for external devices. In particular, interface(s) 1001 may be operative to acquire, from one or more sensors 120 operatively connected to computing device 100-800, nerve signals captured from throat 1 1 1 of user 1 10 subvocalizing at least a part of a displayed presentation 131 of the text. Alternatively, interface(s) 1001 may be operative to acquire, from camera 102, a video sequence of user 1 10 subvocalizing at least a part of presentation 131 . Processing means 1000 further comprises a presentation module 1002, an acquisition module 1003, and a reading parameter module 1004, which are configured to cause computing device 100-800 to perform in accordance with embodiments of the invention as described herein. In particular, presentation module 1002 is configured to display a presentation 131 of the text, acquisition module 1003 is configured to acquire a representation of user 1 10 subvocalizing at least a part of presentation 131 , reading parameter module 1004 is configured to measure one or more reading parameters from the acquired representation, which reading parameters characterize a manner in which user 1 10 is reading
presentation 131 , and presentation module 1002 is further configured to adapt displayed presentation 131 based on the measured reading
parameters.
Acquisition module 1003 may be configured to acquire a
representation of user 1 10 subvocalizing at least a part of presentation 131 by acquiring, from one or more sensors 120 operatively connected to computing device 100-800, nerve signals captured from a throat 1 1 1 of user 1 10 subvocalizing the part of presentation 131 , and deriving the representation of user 1 10 subvocalizing at least a part of presentation 131 text as a representation of the nerve signals. Alternatively, acquisition module 1003 may be configured to acquire a representation of user 1 10 subvocalizing at least a part of presentation 131 by acquiring, from
camera 102 operatively connected to computing device 100-800, a video sequence of user 1 10 subvocalizing the part of presentation 131 , magnifying, by video processing the acquired video sequence, motions of one or more body parts 1 1 1-1 15 of user 1 10, which motions are correlated with the subvocalized part of presentation 131 , and deriving the representation of user 1 10 subvocalizing at least a part of presentation 131 as a representation of the magnified motions.
Reading parameter module 1004 may be configured to measure the one or more reading parameters as one or more of a reading speed, a reading continuity, and a combination thereof.
Presentation module 1002 may further be configured to adapt displayed presentation 131 based on a comparison of the measured reading parameters and one or more corresponding reference reading parameters. Optionally, the one or more reference reading parameters may be
associatively stored with the text.
Reading parameter module 1004 may further be configured to derive the one or more reference reading parameters from acquired representations of user 1 10 subvocalizing text by measuring one or more reading parameters from the acquired representations, which reading parameters characterize a manner in which user 1 10 is reading text, and storing the measured reading parameters as the reference reading parameters. Optionally, the one or more reference reading parameters may be dependent on a type of the text.
Presentation module 1002 may further be configured to adapt displayed presentation 131 further based on a type of the text.
Presentation module 1002 may further be configured to adapt displayed presentation 131 by adding information and/or removing
information.
Presentation module 1002 may further be configured to adapt displayed presentation 131 by providing a summary of at least a part of the text.
Presentation module 1002 may further be configured to adapt displayed presentation 131 by increasing a font size of the displayed text and/or magnifying the displayed text.
Presentation module 1002 may further be configured to adapt displayed presentation 131 of the text by adding one or more references to additional information.
Presentation module 1002 may further be configured to adapt displayed presentation 131 of the text by changing a layout of the presented text.
Interface(s) 901 and 1001 , and modules 1002-1004, as well as any additional modules comprised in processing means 1000, may be
implemented by any kind of electronic circuitry, e.g., any one, or a combination of, analogue electronic circuitry, digital electronic circuitry, and processing circuitry executing a suitable computer program.
In the following, embodiments 1 100 of the method of displaying a text to a user 1 10 of a computing device are described with reference to Fig. 1 1 , in accordance with embodiments of the invention. An embodiment of method 1 100 may be performed by a computing device such as a mobile phone, a mobile terminal, a smartphone, a tablet, a personal computer, a laptop, an e-book reader, a computer display, a television, or a media player. Method 1 100 comprises displaying 1 101 a presentation 131 of the text, acquiring 1 102 a representation of user 1 10 subvocalizing at least a part of presentation 131 , measuring 1 103 one or more reading parameters from the acquired representation, which reading parameters characterize a manner in which user 1 10 is reading presentation 131 , and adapting 1 104 displayed presentation 131 based on the measured reading parameters. The one or more reading parameters may be one or more of a reading speed, a reading continuity, or a combination thereof.
Optionally, adapting 1 104 displayed presentation 131 is based on a comparison of the measured reading parameters and one or more
corresponding reference reading parameters. Further optionally, the one or more reference reading parameters may be associatively stored with the text.
Method 1 100 may further comprise deriving the one or more reference reading parameters from acquired representations of user 1 10 subvocalizing text by measuring one or more reading parameters from the acquired representations, which reading parameters characterize a manner in which user 1 10 is reading text, and storing the measured reading parameters as the reference reading parameters. Optionally, the one or more reference reading parameters may be dependent on a type of the text.
Optionally, adapting 1 104 displayed presentation 131 may further be based on a type of the text. Acquiring 1 102 a representation of user 1 10 subvocalizing at least a part of presentation 131 may comprise acquiring, from one or more sensors 120 operatively connected to the computing device, nerve signals captured from a throat 1 1 1 of user 1 10 subvocalizing the part of
presentation 131 , and deriving the representation of user 1 10 subvocalizing at least a part of presentation 131 as a representation of the nerve signals. Alternatively, acquiring 1 102 a representation of user 1 10 subvocalizing at least a part of presentation 131 may comprise acquiring, from a camera 102 operatively connected to the computing device, a video sequence of user 1 10 subvocalizing the part of presentation 131 , magnifying, by video processing the acquired video sequence, motions of one or more body parts 1 1 1-1 15 of user 1 10, which motions are correlated with the subvocalized part of presentation 131 , and deriving the representation of user 1 10 subvocalizing at least a part of presentation 131 as a representation of the magnified motions.
Optionally, adapting 1 104 displayed presentation 131 may comprise adding information and/or removing information.
Optionally, adapting 1 104 displayed presentation 131 may comprise providing a summary of at least a part of the text.
Optionally, adapting 1 104 displayed presentation 131 may comprise increasing a font size of the displayed text and/or magnifying the displayed text.
Optionally, adapting 1 104 displayed presentation 131 may comprise adding one or more references to additional information.
Optionally, adapting 1 104 displayed presentation 131 may comprise changing a layout of the presented text.
It will be appreciated that method 1 100 may comprise additional, or modified, steps in accordance with what is described throughout this disclosure. An embodiment of method 1 100 may be implemented as software, such as computer program 904, to be executed by a processing unit or processing circuitry comprised in the computing device, whereby the computing device becomes operative to perform in accordance with embodiments of the invention described herein.
The person skilled in the art realizes that the invention by no means is limited to the embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims.

Claims

1 . A computing device (100; 200; 500; 600; 700; 800) for displaying a text to a user (1 10) of the computing device, the computing device
comprising processing (103; 800; 900) means being operative to:
display a presentation (131 ; 531 ) of the text,
acquire a representation of the user subvocalizing at least a part of the presentation of the text,
measure one or more reading parameters from the acquired representation, which reading parameters characterize a manner in which the user is reading the presentation of the text, and
adapt the displayed presentation of the text based on the measured reading parameters.
2. The computing device according to claim 1 , the processing means being operative to acquire a representation of the user subvocalizing at least a part of the presentation of the text by:
acquiring, from one or more sensors (120) operatively connected to the computing device, nerve signals captured from a throat (1 1 1 ) of the user subvocalizing the part of the presentation of the text, and
deriving the representation of the user subvocalizing at least a part of the presentation of the text as a representation of the nerve signals.
3. The computing device according to claim 1 , the processing means being operative to acquire a representation of the user subvocalizing at least a part of the presentation of the text by:
acquiring, from a camera (102) operatively connected to the computing device, a video sequence of the user subvocalizing the part of the presentation of the text, magnifying, by video processing the acquired video sequence, motions of one or more body parts (1 1 1 -1 15) of the user, which motions are correlated with the subvocalized part of the presentation of the text, and deriving the representation of the user subvocalizing at least a part of the presentation of the text as a representation of the magnified motions.
4. The computing device according to any one of claims 1 to 3, wherein the one or more reading parameters are one or more of: a reading speed, a reading continuity, and a combination thereof.
5. The computing device according to any one of claims 1 to 4, the processing means being operative to adapt the displayed presentation of the text based on a comparison of the measured reading parameters and one or more corresponding reference reading parameters.
6. The computing device according to claim 5, wherein the one or more reference reading parameters are associatively stored with the text.
7. The computing device according to claim 5, the processing means being further operative to derive the one or more reference reading parameters from acquired representations of the user subvocalizing text by: measuring one or more reading parameters from the acquired representations, which reading parameters characterize a manner in which the user is reading text, and
storing the measured reading parameters as the reference reading parameters.
8. The computing device according to claim 7, wherein the one or more reference reading parameters are dependent on a type of the text.
9. The computing device according to any one of claims 1 to 8, the processing means being operative to adapt the displayed presentation of the text further based on a type of the text.
10. The computing device (500) according to any one of claims 1 to 9, the processing means being operative to adapt the displayed presentation of the text by at least one of: adding information and removing information.
1 1 . The computing device (600) according to any one of claims 1 to 9, the processing means being operative to adapt the displayed presentation of the text by providing a summary (632) of at least a part of the text.
12. The computing device (700) according to any one of claims 1 to 9, the processing means being operative to adapt the displayed presentation of the text by at least one of: increasing a font size of the displayed text and magnifying the displayed text.
13. The computing device (800) according to any one of claims 1 to 9, the processing means being operative to adapt the displayed presentation of the text by adding one or more references to additional information.
14. The computing device according to any one of claims 1 to 9, the processing means being operative to adapt the displayed presentation of the text by changing a layout of the presented text.
15. The computing device according to any one of claims 1 to 14, being any one of: a mobile phone, a mobile terminal, a smartphone, a tablet, a personal computer, a laptop, an e-book reader, a computer display, a television, and a media player.
16. A method (1 100) of displaying a text to a user (1 10) of a computing device, the method comprising:
displaying (1 101 ) a presentation (131 ) of the text,
acquiring (1 102) a representation of the user subvocalizing at least a part of the presentation of the text,
measuring (1 103) one or more reading parameters from the acquired representation, which reading parameters characterize a manner in which the user is reading the presentation of the text, and
adapting (1 104) the displayed presentation of the text based on the measured reading parameters.
17. The method according to claim 16, wherein the acquiring (1 102) a representation of the user subvocalizing at least a part of the presentation of the text comprises:
acquiring, from one or more sensors (120) operatively connected to the computing device, nerve signals captured from a throat (1 1 1 ) of the user subvocalizing the part of the presentation of the text, and
deriving the representation of the user subvocalizing at least a part of the presentation of the text as a representation of the nerve signals.
18. The method according to claim 16, wherein the acquiring (1 102) a representation of the user subvocalizing at least a part of the presentation of the text comprises:
acquiring, from a camera (102) operatively connected to the computing device, a video sequence of the user subvocalizing the part of the presentation of the text,
magnifying, by video processing the acquired video sequence, motions of one or more body parts (1 1 1 -1 15) of the user, which motions are correlated with the subvocalized part of the presentation of the text, and deriving the representation of the user subvocalizing at least a part of the presentation of the text as a representation of the magnified motions.
19. The method according to any one of claims 16 to 18, wherein the one or more reading parameters are one or more of: a reading speed, a reading continuity, and a combination thereof.
20. The method according to any one of claims 16 to 19, wherein the adapting (1 104) the displayed presentation of the text is based on a comparison of the measured reading parameters and one or more corresponding reference reading parameters.
21 . The method according to claim 20, wherein the one or more reference reading parameters are associatively stored with the text.
22. The method according to claim 20, further comprising deriving the one or more reference reading parameters from acquired representations of the user subvocalizing text by:
measuring one or more reading parameters from the acquired representations, which reading parameters characterize a manner in which the user is reading text, and
storing the measured reading parameters as the reference reading parameters.
23. The method according to claim 22, wherein the one or more reference reading parameters are dependent on a type of the text.
24. The method according to any one of claims 16 to 23, wherein the adapting (1 104) the displayed presentation of the text is further based on a type of the text.
25. The method according to any one of claims 16 to 24, wherein the adapting (1 104) the displayed presentation of the text comprises at least one of: adding information and removing information.
26. The method according to any one of claims 16 to 24, wherein the adapting (1 104) the displayed presentation of the text comprises providing a summary of at least a part of the text.
27. The method according to any one of claims 16 to 24, wherein the adapting (1 104) the displayed presentation of the text comprises at least one of: increasing a font size of the displayed text and magnifying the displayed text.
28. The method according to any one of claims 16 to 24, wherein the adapting (1 104) the displayed presentation of the text comprises adding one or more references to additional information.
29. The method according to any one of claims 16 to 24, wherein the adapting (1 104) the displayed presentation of the text comprises changing a layout of the presented text.
30. A computer program (904) comprising computer-executable instructions for causing a device to perform the method according to any one of claims 16 to 29, when the computer-executable instructions are executed on a processing unit (902) comprised in the device.
31 . A computer program product comprising a computer-readable storage medium (903), the computer-readable storage medium having the computer program (904) according to claim 30 embodied therein.
PCT/EP2017/050452 2017-01-11 2017-01-11 Displaying text to a user of a computing device WO2018130273A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/EP2017/050452 WO2018130273A1 (en) 2017-01-11 2017-01-11 Displaying text to a user of a computing device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2017/050452 WO2018130273A1 (en) 2017-01-11 2017-01-11 Displaying text to a user of a computing device

Publications (1)

Publication Number Publication Date
WO2018130273A1 true WO2018130273A1 (en) 2018-07-19

Family

ID=57821959

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2017/050452 WO2018130273A1 (en) 2017-01-11 2017-01-11 Displaying text to a user of a computing device

Country Status (1)

Country Link
WO (1) WO2018130273A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024023459A1 (en) * 2022-07-28 2024-02-01 Universite Claude Bernard Lyon 1 Device and method for measuring the speed of a user in decrypting an item of visual information

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11353468A (en) * 1998-06-11 1999-12-24 Nippon Hoso Kyokai <Nhk> System, method, and recording medium for speaking speed measurement
GB2501298A (en) * 2012-04-19 2013-10-23 Ibm Approximating electronic document last reading position
US20160378720A1 (en) * 2015-06-29 2016-12-29 Amazon Technologies, Inc. Dynamic adjustment of rendering parameters to optimize reading speed

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11353468A (en) * 1998-06-11 1999-12-24 Nippon Hoso Kyokai <Nhk> System, method, and recording medium for speaking speed measurement
GB2501298A (en) * 2012-04-19 2013-10-23 Ibm Approximating electronic document last reading position
US20160378720A1 (en) * 2015-06-29 2016-12-29 Amazon Technologies, Inc. Dynamic adjustment of rendering parameters to optimize reading speed

Non-Patent Citations (9)

* Cited by examiner, † Cited by third party
Title
CHRIS PARNIN: "Subvocalization - Toward Hearing the Inner Thoughts of Developers", PROGRAM COMPREHENSION (ICPC), 2011 IEEE 19TH INTERNATIONAL CONFERENCE ON, IEEE, 22 June 2011 (2011-06-22), pages 197 - 200, XP031912437, ISBN: 978-1-61284-308-7, DOI: 10.1109/ICPC.2011.49 *
H.-Y. WU; M. RUBINSTEIN; E. SHIH; J. GUTTAG; F. DURAND; W. FREEMAN: "Proceedings of ACM SIGGRAPH", vol. 31, 2012, ACM, article "Eulerian Video Magnification for Revealing Subtle Changes in the World"
I. MOHOMED; J. C. CAI; E. DE LARA: "Proceedings of the 1 st ACM SIGOPS/EuroSys European Conference on Computer Systems 2006, EuroSys '06", 2006, ACM, article "URICA: Usage-awaRe Interactive Content Adaptation for Mobile Devices", pages: 345 - 358
I. MOHOMED; J. C. CAI; S. CHAVOSHI; E. DE LARA: "Proceedings of the 4th international conference on Mobile systems, applications and services, MobiSys '06", 2006, ACM, article "Context-Aware Interactive Content Adaptation", pages: 42 - 55
L. COPELAND; T. GEDEON; S. MENDIS: "Artificial Intelligence Research", vol. 3, 2014, SCIEDU PRESS, article "Predicting reading comprehension scores from eye movements using artificial neural networks and fuzzy output error", pages: 35 - 48
LOPEZ-LARRAZ E ET AL: "Syllable-based speech recognition using EMG", 2010 ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY : (EMBC 2010) ; BUENOS AIRES, ARGENTINA, 31 AUGUST - 4 SEPTEMBER 2010, IEEE, PISCATAWAY, NJ, USA, 31 August 2010 (2010-08-31), pages 4699 - 4702, XP032108447, ISBN: 978-1-4244-4123-5, DOI: 10.1109/IEMBS.2010.5626426 *
M. A. ELGHARIB; M. HEFEEDA; F. DURAND; W. T. FREEMAN: "2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR", 2015, IEEE COMPUTER SOCIETY, article "Video magnification in presence of large motions", pages: 4119 - 4127
N. WADHWA; M. RUBINSTEIN; F. DURAND; W. T. FREEMAN: "SIGGRAPH 2013 Conference Proceedings, ACM Transactions on Graphics", vol. 32, 2013, ACM, article "Phase-based video motion processing"
S. XU; H. JIANG; F. C. M. LAU: "Proceedings of the 14th international conference on Intelligent user interfaces, IUI '09", 2009, ACM, article "User-Oriented Document Summarization through Vision-Based Eye-Tracking", pages: 7 - 16

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024023459A1 (en) * 2022-07-28 2024-02-01 Universite Claude Bernard Lyon 1 Device and method for measuring the speed of a user in decrypting an item of visual information
FR3138291A1 (en) * 2022-07-28 2024-02-02 Universite Claude Bernard Lyon 1 device and method for measuring the speed of a user in deciphering visual information

Similar Documents

Publication Publication Date Title
US10433052B2 (en) System and method for identifying speech prosody
US9031293B2 (en) Multi-modal sensor based emotion recognition and emotional interface
US11837249B2 (en) Visually presenting auditory information
US10902743B2 (en) Gesture recognition and communication
US20140212854A1 (en) Multi-modal modeling of temporal interaction sequences
US11636859B2 (en) Transcription summary presentation
JP2012014394A (en) User instruction acquisition device, user instruction acquisition program and television receiver
Madhuri et al. Vision-based sign language translation device
JP2012059107A (en) Emotion estimation device, emotion estimation method and program
CN112016367A (en) Emotion recognition system and method and electronic equipment
US9028255B2 (en) Method and system for acquisition of literacy
CN111353366A (en) Emotion detection method and device and electronic equipment
US20210201696A1 (en) Automated speech coaching systems and methods
CN111149172B (en) Emotion management method, device and computer-readable storage medium
CN113556603B (en) Method and device for adjusting video playing effect and electronic equipment
EP2632158A1 (en) Method and apparatus for processing information of image including a face
CN111027358B (en) Dictation and newspaper reading method based on writing progress and electronic equipment
WO2018130273A1 (en) Displaying text to a user of a computing device
Zhang et al. A survey on mobile affective computing
CN111026872B (en) Associated dictation method and electronic equipment
Esposito et al. The new Italian audio and video emotional database
US20240055014A1 (en) Visualizing Auditory Content for Accessibility
US20210390167A1 (en) Authenticating a user subvocalizing a displayed text
KR102338684B1 (en) Data process method for prediction of involuntary emotion
Yau Video analysis of mouth movement using motion templates for computer-based lip-reading

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17700505

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17700505

Country of ref document: EP

Kind code of ref document: A1