WO2020046346A1 - Computing device attention determination - Google Patents

Computing device attention determination Download PDF

Info

Publication number
WO2020046346A1
WO2020046346A1 PCT/US2018/048880 US2018048880W WO2020046346A1 WO 2020046346 A1 WO2020046346 A1 WO 2020046346A1 US 2018048880 W US2018048880 W US 2018048880W WO 2020046346 A1 WO2020046346 A1 WO 2020046346A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
images
computing device
paying attention
attention
Prior art date
Application number
PCT/US2018/048880
Other languages
French (fr)
Inventor
King Sui KEI
Alexander Wayne CLARK
Original Assignee
Hewlett-Packard Development Company, L.P.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hewlett-Packard Development Company, L.P. filed Critical Hewlett-Packard Development Company, L.P.
Priority to EP18932010.4A priority Critical patent/EP3759650A4/en
Priority to CN201880092960.1A priority patent/CN112005248A/en
Priority to US17/049,193 priority patent/US12002287B2/en
Priority to PCT/US2018/048880 priority patent/WO2020046346A1/en
Publication of WO2020046346A1 publication Critical patent/WO2020046346A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304—Detection arrangements using opto-electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/193—Preprocessing; Feature extraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition

Definitions

  • Computing devices are used in many scenarios in modern society. Businesses, home users, and other entities use computing devices such as desktop computers, laptops, and tablets to provide information to users of the computing devices.
  • computing devices are used within an educational setting.
  • an instructor or administrator may prepare or acquire information to be presented to students on computing devices.
  • a multimedia presentation may be uploaded to a computing network. Students may then access the network to view the multimedia presentation on their own particular computing device.
  • FIG. 1 is a block diagram of a system for computing device attention determination, according to an example of the principles described herein.
  • FIG. 2 is a flow chart of a method for computing device attention determination, according to an example of the principles described herein.
  • Fig. 3 is a block diagram of a system for computing device attention determination, according to another example of the principles described herein.
  • FIG. 4 is a flow chart of a method for computing device attention determination, according to another example of the principles described herein.
  • FIG. 5 is a diagram of an environment for computing device attention determination, according to an example of the principles described herein.
  • FIG. 6 is a diagram of a machine-readable storage medium for computing device attention determination, according to an example of the principles described herein.
  • computing devices have clearly enhanced information dissemination in a variety of environments, some scenarios may reduce their positive effect. For example, in an educational setting, students may get distracted when in front of a computer screen. Moreover, a teacher in a room full of students in front of computer screens may have a difficult time keeping each student’s focus on their respective computer screens. In attempt to maintain the focus of the students, the teacher may walk around a room and ensure each student is paying attention. However, doing so may be ineffective, especially in a large room full of multiple students and takes an instructors attentions away from other students while individually addressing a particular student. In other settings, distraction from information presented on a computing device is also an issue.
  • a notice may be provided, either to an administrator such as the instructor or a user such as a student, notifying of the lack of focus.
  • the attention analysis system includes a database.
  • the database includes images indicative that users are paying attention to corresponding computing devices and images indicative that users are not paying attention to corresponding computing devices.
  • the attention analysis system also includes an analyzer to extract multiple user position characteristics from a stream of images of a user disposed in front of a computing device.
  • a comparer of the attention analysis system determines, by comparing the stream of images of the user to the images in the database, whether the user is paying attention to the computing device.
  • Multiple user position characteristics are extracted from the stream of images.
  • the stream of images is then compared to a database that includes 1 ) images indicative that users are paying attention to corresponding computing devices and 2) images indicative that users are not paying attention to corresponding computing devices. It is then determined, based on the comparison, whether the user is paying attention to the computing device.
  • the present specification also describes a tangible machine- readable storage medium encoded with instructions executable by a processor.
  • the machine-readable storage medium includes instructions to collect from multiple computing devices, a corpus of images that include 1 ) images indicative that users are paying attention to corresponding computing devices and 2) images indicative that users are not paying attention to corresponding computing devices.
  • the machine-readable storage medium also includes instructions to collect from a computing device, a stream of images of a user disposed in front of the computing device; extract multiple user position characteristics from the stream of images; and compare the stream of images to the corpus of images.
  • the machine-readable storage medium also includes instructions to collect information regarding a user interaction with the computing device; instructions to, based on a comparison of the stream of images to the corpus of images and the information regarding the user interaction with the computing device, determine whether the user is paying attention to the computing device; and instructions to provide notice regarding whether the user is paying attention to the computing device.
  • Fig. 1 is a block diagram of a system (100) for computing device attention determination, according to an example of the principles described herein.
  • the attention analysis system (100) may be used to determine whether a user disposed in front of a computing device is actively paying attention, or focusing, on the material being presented on the associated computing device.
  • the attention analysis system (100) may be presented in any number of environments.
  • One particular example is an educational setting, where a teacher, who may be remote or physically within the same room as the audience, is presenting educational material to students.
  • the presented material may be streamed to those computing devices from a computing device associated with the instructor.
  • an administrator or instructor is monitoring the users as they are independently viewing material on the computing devices.
  • the attention analysis system (100) may be used in other settings such as a business presentation setting. As the attention analysis system (100) is used to determine whether users are paying attention to their respective computing devices, the attention analysis system (100) may include a network connection to the multiple computing devices.
  • the attention analysis system (100) also includes an analyzer (104) to extract multiple user position characteristics from a stream of images of a user disposed in front of a computing device. That is, the analyzer (104) extracts more than just an eye gaze from the stream of images, but also other user position characteristics as described above, which may include mouth position, mouth movement, nose movement, eyebrow/eyelid position, and/or movement as well as general head position/movement.
  • the analyzer (104) may also extract data relating to these characteristic from a number of predefined images which, as described above, are analyzed to form a basis against which the stream of images of the particular user are analyzed.
  • the attention analysis system (100) is a machine-learning system. Being a machine-learning system (100), the system (100) and particularly the analyzer (104) can learn over time, simply by analyzing a variety of images/videos that indicate user attention or lack of user attention. That is, there may be various characteristics that indicate a user is paying attention to a computing device.
  • One specific example is eye gaze.
  • a computing device may include an integrated camera.
  • the position of the user’s eyes towards the computing device screen is indicia that a user is paying attention to the display screen.
  • a user’s eyes are diverted away from the computer screen, i.e., down looking at a mobile device or to the side looking out a window or at another student’s desk/screen, this is indicative that the user is not paying attention.
  • pupil dilation and/or eyelid position For example, if a user’s eyelids are in a more closed position rather than fully open, this may indicate that a user is falling asleep and therefore not paying attention.
  • Other information may be used as well. That is, while specific reference is made to eye information and eye position information, other information may be indicative of a user paying attention. Examples include user head position information and user body position information. For example, similar to the position of the eyes, the position of a user’s head towards the computing device screen may be indicative of paying attention to what is presented on the computing device, while a head angled downward or to the side indicates that the user is looking at something other than the user’s own computing device.
  • the database (102) may include a corpus of images that have tags associated with them that indicate whether the related image identifies a user paying attention or a user not paying attention to the respective computing device.
  • the analyzer (104) can then analyze the information, i.e., eye position, head position, and/or body position, to determine criteria that can be used as a benchmark to determine when another image indicates a user is paying attention to a computing device screen or not. That is, as an
  • the image/video to be analyzed is received, it is compared to determine what and how many characteristics it shares with the images in the database (102). If the image/video to be analyzed shares more characteristics with images indicative of a user paying attention, it may be determined that the user in the image/video to be analyzed is paying attention to the computing device. By comparison, if the image/video to be analyzed shares more characteristics with images indicative of a user not paying attention, it may be determined that the user in the image/video is not paying attention to the computing device.
  • the machine-learning database (102) analyzes multiple characteristics, not just one, of the corpus of images. That is, multiple characteristics, not just one, are used to determine whether a user is paying attention to a computing screen or not.
  • a comparer (106) determines, by comparing the stream of images with the images in the database, whether the user is paying attention to the computing device. That is, the comparer (106) can determine whether the stream of images includes user position characteristics more consistent with images indicative of paying attention or with images indicative of a lack of attention. Based on such a comparison, a determination is made regarding user attention, and a notification may be presented to a user. For example, a notice may be provided to a user of the computing device reminding them to return their attention to the computing device. A notice may also be provided to an administrator of the environment such that remedial action may be taken. For example, in the case of a classroom setting, an output of each student’s level of attention may be provided to an instructor such that the instructor may take remedial action to bring the student’s attention back to the computing devices.
  • an instructor may be provided with a management console that can gauge the state of the classroom based on its attention levels. On an individual level, the instructor is notified that a student may be having a tough time focusing. This can give the instructor an opportunity to ask the student if they need help, to find a new assignment to re- interest the student, or to help the instructor control a situation.
  • notifications to the instructor management console can alert the instructor quickly such that they may assess the situation.
  • the stream of images/video of a user is collected from an integrated camera on a user’s computing device.
  • a laptop computer or desktop may include a camera that can capture the images from which user position characteristics are extracted.
  • a separate camera such as one in the classroom may be used to determine student attention.
  • an attention analysis system (100) provides a simply way to determine whether a single user, or multiple users, are focusing on a computing device screen. This may include analyzing more information than just eye gaze. That is other user position information such as head position, body position, and eye position may be relied on. Relying on more than just eye gaze provides a more accurate determination as to attention. That is, eye gaze alone is a single data point and may not accurately describe whether a user is paying attention or not. In a specific example, a user’s eyes may be directed towards a computer screen. However, the user’s eyelids may be partially closed indicating a user is not paying attention. Accordingly, the system (100) by relying on multiple user position characteristics provides a more accurate determination of user attention to a computing device screen.
  • Fig. 2 is a flow chart of a method (200) for computing device attention determination, according to an example of the principles described herein.
  • a stream of images is collected (block 201 ) of a user in front of a computing device.
  • the stream of images may be still images or may be images strung together as frames of a video clip.
  • the images may be collected (block 201 ) from a camera integrated into the computing device of the user or may be collected from another source such as a camera(s) disposed in the room with the user. In either case, the camera captures images of the user as they are sitting in front of a respective computing device.
  • user position characteristics are extracted (block 202) from the stream of images.
  • Such user position characteristics include eye gaze and additional characteristics.
  • user eye information may include additional information relating to a position/movement of a user’s eye.
  • user eye information may include a gaze direction, pupil dilation, eye movement, etc. Additional pieces of eye position information provide better results regarding user attention. That is, while an eye gaze towards a user computing device is indicia of user attention, pupil dilation may be also be indicative of user attention. As yet another example eye movement may be indicative of user attention. The combination of this information provides more information than eye gaze alone regarding user attention.
  • Eye gaze alone may indicate that this user is not paying attention.
  • knowing that the computing device includes a keyboard eye movement information may indicate that a user’s eyes are frequently going from a downward position (looking at the keyboard) to an upward position (looking at the screen).
  • eye movement information may indicate that a user’s eyes are frequently going from a downward position (looking at the keyboard) to an upward position (looking at the screen).
  • a user’s gaze is not entirely at the computing device, they may still be paying attention, albeit with their gaze in a different direction, i.e., at the keyboard or looking at a text book.
  • the attention analysis system may extract (block 202) additional characteristics, in addition to eye position information.
  • user head position information that indicates a head position/movement may also indicate user attention. Examples of such head position/movement may indicate that a user is not looking at the computing device screen. For example, a head position angled away from the computing device screen indicates a user is not paying attention.
  • Head position information may also be used to expound upon information collected eye position information. For example, a user may rest their head on their hand at an angle while looking at the computing device screen. Eye gaze alone may therefore indicate a user is paying attention, as their eyes are in the direction of the computing screen. However, the resting of the head on the propped hand of a user is indicia that a user, despite looking at the screen, is not paying attention.
  • head position information include head feature position information, such as a position/movement of eyelids, eyebrows, nose, and/or mouth.
  • head feature position information such as a position/movement of eyelids, eyebrows, nose, and/or mouth.
  • droopy eyelids may indicate that again, while a user gaze may be towards a computing device screen, a user is in fact not paying attention.
  • the present specification by relying on user position information beyond eye gaze provides more details and more data points such that a more accurate determination of whether a user is paying attention or not may be made.
  • the information may include user body position information.
  • shoulder angle information may be indicative of whether a user is slouching in their chair, which may be indicia that a user is not paying attention.
  • body position characteristics of a user While specific reference is made to particular body position characteristics of a user, other body position/movement characteristics may be extracted (block 202) from a stream of images for a user by which it may be determined whether that user is paying attention or not.
  • the stream of images, and more particularly the extracted characteristics from the stream of images, are compared (block 203) against a database (Fig. 1 , 102) of images.
  • the database (Fig. 1 , 102) of images against which the stream of images are compared have been similarly analyzed with characteristics extracted therefrom.
  • the database (Fig. 1 , 102) of images includes a training set which includes images and a classification as to whether it indicates a user is paying attention or not.
  • the features of these images are analyzed such that user position characteristics may be mapped to either an attentive or non-attentive user.
  • the comparison determines whether the stream of images shares more common characteristics with images reflecting high attention or with images reflecting poor attention.
  • the comparison (block 203) of the stream of images may be performed over a period of time. For example, it may not be the case that a user looking away from his/her computing device screen for a few seconds indicates they are not paying attention. However, the user looking away from his/her computing device for 10-15 seconds may be more indicative of a lack of attention. That is, distraction is gauged in combination with the length of time that a user’s face is positioned away from the screen and the number of occurrences. Outliers and anomalies are considered momentary pauses and rests. Recurring infractions by comparison, provide a more sound indication of user distraction.
  • Such a determination (block 204) may be made based on a degree of similarity between the stream of images and images of the different subsets in the database (Fig. 1 , 102). That is, the attention analysis system (Fig. 1 , 100) determines with which of the subset of images the stream of images has more in common. That is, whether the stream of images has more in common with the images indicative that a user is paying attention to a corresponding computing device or whether the stream of images has more in common with the images indicative that a user is not paying attention to a corresponding computing device.
  • the term“in common” refers to a quantity of shared user position characteristics of the images.
  • the present specification describes a method (200) that relies on a reliable set of images, i.e. the training set which may include thousands or even millions of examples, from which characteristics indicative of attention or distraction are drawn. Comparing a set of images depicting a particular user against these images allows for a determination as to whether a user is paying attention to their computing device.
  • Fig. 3 is a block diagram of a system (100) for computing device attention determination, according to another example of the principles described herein.
  • the attention analysis system (100) includes a database (102), an analyzer (104), and a comparer (106).
  • the attention analysis system (100) includes additional components.
  • the attention analysis system (100) includes an aggregator (308) to generate an output resulting from the analysis of different sets of images, each set corresponding to a different user. That is, the aggregator (308) generates an output based on comparisons of multiple streams of images relating to multiple users with the images in the database (102). In so doing, an aggregate level of attention of multiple users can be indicated. That is, a different set of remedial actions may be carried out if multiple users are not paying attention as opposed to a single user not paying attention.
  • the information collected by the aggregator (308) may also be used to evaluate a presenter, teacher, or administrator. For example, if a majority of the audience is not paying attention to a particular presentation, as indicated by a comparison of the stream of images for those users against the database of images, it may indicate that the presenter, teacher, or administrator has not effectively presented the material. By comparison, if a majority of users are actively engaged in the presentation, it may indicate that the presenter, teacher, or administrator has been effective in delivery the content.
  • the attention analysis system (100) includes an identifier (310) to identify patterns in the aggregate level of attention. For example, it may be the case that a certain group of users are not paying attention, but that others are.
  • the identifier (310) may identify such patterns.
  • the identifier (310) may indicate that users within a certain demographic, for example a certain age range, are more prone to falling out of attention as compared to another age range. Accordingly, this may be presented to the administrator to tailor the subject matter to be more wide-reaching in regards to audience demographics.
  • the additional components within the attention analysis system (100) provide additional information that can be used to improve user focus.
  • the information collected could be transmitted elsewhere.
  • an aggregate level of attention may be passed such that an evaluation of the presenter or instructor may be made.
  • the aggregate level of attention as well as the output of the identifier (310) may be passed to enhance and alter the subject matter, or form, of a presentation. For example, it may be the case that a video presentation shown in a dark room leads to many users falling asleep as indicated by a comparison of body position features of the various users compared against images in the database. Accordingly, the presentation of the material may be changed to an interactive multimedia presentation rather than a video display.
  • Fig. 4 is a flow chart of a method (400) for computing device attention determination, according to another example of the principles described herein.
  • a stream of images of a user in front of a computing device is collected (block 401 ) and user position characteristics are extracted (block 402) therefrom.
  • This stream of images, and more particularly the extracted user position characteristics are compared (block 403) against images in a training database.
  • the attention analysis system may rely on other information as well. That is, while user position information may provide indicia of user attention, additional information may provide a more accurate determination. For example, a user may be looking at a computing device, while the user’s mind may be elsewhere. Thus, eye gaze information alone may indicate a user is paying attention, however additional criteria may indicate that the user is not paying attention to the computer screen that they are looking at.
  • the attention analysis system may rely on other pieces of information, specifically user interaction with the computing device.
  • user interaction with the computing device may include a user manipulation of a user input device.
  • a user may move a mouse cursor during an interactive display, or may enter information via a keyboard. This interaction may indicate attention to the presentation, regardless of an eye gaze direction.
  • multiple other types of user input devices may exist and a user’s interaction with those devices may be collected (block 404).
  • user interaction may be indicated by a change of information displayed on the computing device.
  • a multimedia presentation may be displayed on the computing device.
  • the attention analysis system (Fig. 1 , 100) may detect a change in display content for example as a user advances through slides of the presentation. Such a change in displayed content can be considered in addition to the comparison (block 403) of images with the database to determine whether or not the user is actively paying attention to the computing device.
  • noise level information around the computing device may be collected (block 405). That is, ambient noise in a space can be a source of distraction to individuals. Accordingly, a microphone on a computing device or in a room may be used to determine a level of ambient noise around the user. This information, in addition to user position characteristics and user interaction information can provide a more accurate view of user focus.
  • the attention analysis system Fig. 1 , 100
  • the present method (400) provides an environment where multiple pieces of different types of information, not just related to the user movement/position information, are used in determining user attention.
  • the attention analysis system Fig. 1 , 100
  • the computing device in which it is installed can provide a more accurate picture regarding user attention such that more targeted remedial measures, if needed, may be executed.
  • Fig. 5 is a diagram of an environment for computing device (512) attention determination, according to an example of the principles described herein.
  • the attention analysis system (100) is coupled to a network of computing devices (512).
  • One of those computing devices (512-1 ) is associated with a user of whom it is desired to know if he/she is paying attention to the computing device (512-1 ).
  • the other computing devices (512-2, 512-3, 512-4, 512-5) provide the training set of images that are included in the database (Fig. 1 , 102) to which the stream of images from the computing device (512-1 ) are compared.
  • the attention analysis system (100) also collects streams of images (514) from the first computing device (512-1 ).
  • the images (514) may be still images taken over a period of time.
  • the images (514) may be frames of a video clip also taken over a period of time.
  • the computing device (512) also transmits other information (516) such as information indicating a user interaction with the computing device (512- 1 ) and noise levels around the computing device (512-1 ). All this information is processed by the attention analysis system (100) to determine whether a user is paying attention. Such an analysis may be performed for any number of computing devices (512). That is, such an analysis may be done for a single computing device (512) to ascertain the attention level of one user or such analysis may be done for multiple computing devices (512) to ascertain the attention level of multiple users.
  • the information is then output by the attention analysis system (100). That is, the information may be output via a
  • the management console wherein a user of the management console can address individual distraction or aggregate distraction.
  • the information is transmitted further, for example to a server, where it is
  • Fig. 6 is a diagram of a machine-readable storage medium (618) for computing device attention determination, according to an example of the principles described herein.
  • a computing system includes various hardware components. Specifically, the computing system includes a processor.
  • Machine-readable storage medium (618) is communicatively coupled to the processor.
  • the machine-readable storage medium (618) includes a number of instruction sets (620, 622, 624, 626, 628, 630, 632) for performing a designated function.
  • the machine-readable storage medium (618) causes the processor to execute the designated function of the instruction sets (620, 622, 624, 626, 628, 630, 632).
  • the instruction sets (620, 622, 624, 626, 628, 630, 632) may be distributed (e.g., stored) across multiple machine-readable storage mediums.
  • the machine-readable storage medium (618) represents any tangible and non-transitory memory capable of storing data such as
  • Compare instructions (626), when executed by a processor, may cause the computing system to compare the stream of images (Fig. 5, 514) to the corpus of images.
  • Interaction instructions (628), when executed by a processor, may cause the computing system to collect information regarding a user interaction with the computing device (Fig. 5, 512-1 ).
  • Determine instructions (630), when executed by a processor, may cause the computing system to, based on a comparison of the stream of images (Fig. 5, 514) to the corpus of images and the information regarding the user interaction with the computing device (Fig. 5, 512-1 ), determine whether the user is paying attention to the computing device (Fig. 5, 512-1 ).
  • an attention determining system 1 allows for a systematic and effective way to track focus to material being presented on a computing device; 2) allows remote presenters information regarding audience engagement; and 3) allows presenters to focus on preparation and presentation of material without worrying about audience distraction.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Ophthalmology & Optometry (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

In one example in accordance with the present disclosure, an attention analysis system is described. The attention analysis system includes a database. The database includes 1) images indicative that users are paying attention to corresponding computing devices and 2) images indicative that users are not paying attention to corresponding computing devices. The system also includes an analyzer to extract multiple user position characteristics from a stream of images of a user disposed in front of a computing device. A comparer of the system determines, by comparing the stream of images of the user to the images in the database, whether the user is paying attention to the computing device.

Description

COMPUTING DEVICE ATTENTION DETERMINATION
BACKGROUND
[0001] Computing devices are used in many scenarios in modern society. Businesses, home users, and other entities use computing devices such as desktop computers, laptops, and tablets to provide information to users of the computing devices. As a specific example, computing devices are used within an educational setting. In these examples, an instructor or administrator may prepare or acquire information to be presented to students on computing devices. For example, a multimedia presentation may be uploaded to a computing network. Students may then access the network to view the multimedia presentation on their own particular computing device.
BRIEF DESCRIPTION OF THE DRAWINGS
[0002] The accompanying drawings illustrate various examples of the principles described herein and are part of the specification. The illustrated examples are given merely for illustration, and do not limit the scope of the claims.
[0003] Fig. 1 is a block diagram of a system for computing device attention determination, according to an example of the principles described herein.
[0004] Fig. 2 is a flow chart of a method for computing device attention determination, according to an example of the principles described herein. [0005] Fig. 3 is a block diagram of a system for computing device attention determination, according to another example of the principles described herein.
[0006] Fig. 4 is a flow chart of a method for computing device attention determination, according to another example of the principles described herein.
[0007] Fig. 5 is a diagram of an environment for computing device attention determination, according to an example of the principles described herein.
[0008] Fig. 6 is a diagram of a machine-readable storage medium for computing device attention determination, according to an example of the principles described herein.
[0009] Throughout the drawings, identical reference numbers designate similar, but not necessarily identical, elements. The figures are not necessarily to scale, and the size of some parts may be exaggerated to more clearly illustrate the example shown. Moreover, the drawings provide examples and/or implementations consistent with the description; however, the description is not limited to the examples and/or implementations provided in the drawings.
DETAILED DESCRIPTION
[0010] While computing devices have clearly enhanced information dissemination in a variety of environments, some scenarios may reduce their positive effect. For example, in an educational setting, students may get distracted when in front of a computer screen. Moreover, a teacher in a room full of students in front of computer screens may have a difficult time keeping each student’s focus on their respective computer screens. In attempt to maintain the focus of the students, the teacher may walk around a room and ensure each student is paying attention. However, doing so may be ineffective, especially in a large room full of multiple students and takes an instructors attentions away from other students while individually addressing a particular student. In other settings, distraction from information presented on a computing device is also an issue. [0011] Accordingly, the present specification describes a method of addressing user distraction in front of a computing device screen. For example, as described above, one teacher in a classroom of many students may not be able to pay full attention to all the students and make sure that each student is actively engaged in the learning process. Accordingly, the present specification addresses this issue by using image processing to detect a facing direction of a student’s head with a camera - sometimes disposed on the computing device - and sending a notification to an instructor management console when a student is not paying attention, i.e., has been facing away from the computing device screen for a certain period. In addition to eye gaze, other user position information may be relayed to the management console along with other non- user position information, such as a user’s interaction with the computing device. When it is determined that a user is not paying attention to their computing device, a notice may be provided, either to an administrator such as the instructor or a user such as a student, notifying of the lack of focus.
[0012] Specifically, the present specification describes an attention analysis system. The attention analysis system includes a database. The database includes images indicative that users are paying attention to corresponding computing devices and images indicative that users are not paying attention to corresponding computing devices. The attention analysis system also includes an analyzer to extract multiple user position characteristics from a stream of images of a user disposed in front of a computing device. A comparer of the attention analysis system determines, by comparing the stream of images of the user to the images in the database, whether the user is paying attention to the computing device.
[0013] The present specification also describes a method. According to the method, a stream of images is collected from a computing device. The stream of images depict a user disposed in front of the computing device.
Multiple user position characteristics are extracted from the stream of images. The stream of images is then compared to a database that includes 1 ) images indicative that users are paying attention to corresponding computing devices and 2) images indicative that users are not paying attention to corresponding computing devices. It is then determined, based on the comparison, whether the user is paying attention to the computing device.
[0014] The present specification also describes a tangible machine- readable storage medium encoded with instructions executable by a processor. The machine-readable storage medium includes instructions to collect from multiple computing devices, a corpus of images that include 1 ) images indicative that users are paying attention to corresponding computing devices and 2) images indicative that users are not paying attention to corresponding computing devices. The machine-readable storage medium also includes instructions to collect from a computing device, a stream of images of a user disposed in front of the computing device; extract multiple user position characteristics from the stream of images; and compare the stream of images to the corpus of images. The machine-readable storage medium also includes instructions to collect information regarding a user interaction with the computing device; instructions to, based on a comparison of the stream of images to the corpus of images and the information regarding the user interaction with the computing device, determine whether the user is paying attention to the computing device; and instructions to provide notice regarding whether the user is paying attention to the computing device.
[0015] In summary, using such an attention determining system 1 ) allows for a systematic and effective way to track focus to material being presented on a computing device; 2) allows remote presenters information regarding audience engagement; and 3) allows presenters to focus on preparation and presentation of material without worrying about audience distraction.
[0016] Turning now to the figures, Fig. 1 is a block diagram of a system (100) for computing device attention determination, according to an example of the principles described herein. Specifically, the attention analysis system (100) may be used to determine whether a user disposed in front of a computing device is actively paying attention, or focusing, on the material being presented on the associated computing device. The attention analysis system (100) may be presented in any number of environments. One particular example is an educational setting, where a teacher, who may be remote or physically within the same room as the audience, is presenting educational material to students. The presented material may be streamed to those computing devices from a computing device associated with the instructor. In another example, an administrator or instructor is monitoring the users as they are independently viewing material on the computing devices. While specific reference is made to an educational setting, the attention analysis system (100) may be used in other settings such as a business presentation setting. As the attention analysis system (100) is used to determine whether users are paying attention to their respective computing devices, the attention analysis system (100) may include a network connection to the multiple computing devices.
[0017] The attention analysis system (100) includes a database (102). This database (102) includes a corpus of images. The images may be categorized into at least two subsets of images. A first being images that are indicative of users paying attention to corresponding computing devices and a second being images that are indicative of users not paying attention to corresponding computing devices. Throughout the specification, the term “images” refers to still images or frames of a video. These images serve as a baseline for determining when a particular user is paying attention to their respective computing device. What types of images indicate a person showing attention are gathered in the model is a process called training. This training is often a manual exercise, wherein a user makes the assessment whether or not an image indicates that a person is paying attention or not. For every image that is trained, the user makes a judgement and assigns a positive or negative value. It is this value that can be used to create the neural net and predict future images.
[0018] The attention analysis system (100) also includes an analyzer (104) to extract multiple user position characteristics from a stream of images of a user disposed in front of a computing device. That is, the analyzer (104) extracts more than just an eye gaze from the stream of images, but also other user position characteristics as described above, which may include mouth position, mouth movement, nose movement, eyebrow/eyelid position, and/or movement as well as general head position/movement. The analyzer (104) may also extract data relating to these characteristic from a number of predefined images which, as described above, are analyzed to form a basis against which the stream of images of the particular user are analyzed.
[0019] In some examples, the attention analysis system (100) is a machine-learning system. Being a machine-learning system (100), the system (100) and particularly the analyzer (104) can learn over time, simply by analyzing a variety of images/videos that indicate user attention or lack of user attention. That is, there may be various characteristics that indicate a user is paying attention to a computing device. One specific example is eye gaze.
That is, a computing device may include an integrated camera. The position of the user’s eyes towards the computing device screen is indicia that a user is paying attention to the display screen. By comparison, when a user’s eyes are diverted away from the computer screen, i.e., down looking at a mobile device or to the side looking out a window or at another student’s desk/screen, this is indicative that the user is not paying attention.
[0020] Another example is pupil dilation and/or eyelid position. For example, if a user’s eyelids are in a more closed position rather than fully open, this may indicate that a user is falling asleep and therefore not paying attention. Other information may be used as well. That is, while specific reference is made to eye information and eye position information, other information may be indicative of a user paying attention. Examples include user head position information and user body position information. For example, similar to the position of the eyes, the position of a user’s head towards the computing device screen may be indicative of paying attention to what is presented on the computing device, while a head angled downward or to the side indicates that the user is looking at something other than the user’s own computing device.
[0021] Accordingly, the database (102) may include a corpus of images that have tags associated with them that indicate whether the related image identifies a user paying attention or a user not paying attention to the respective computing device. The analyzer (104) can then analyze the information, i.e., eye position, head position, and/or body position, to determine criteria that can be used as a benchmark to determine when another image indicates a user is paying attention to a computing device screen or not. That is, as an
image/video to be analyzed is received, it is compared to determine what and how many characteristics it shares with the images in the database (102). If the image/video to be analyzed shares more characteristics with images indicative of a user paying attention, it may be determined that the user in the image/video to be analyzed is paying attention to the computing device. By comparison, if the image/video to be analyzed shares more characteristics with images indicative of a user not paying attention, it may be determined that the user in the image/video is not paying attention to the computing device. Thus, the machine-learning database (102) analyzes multiple characteristics, not just one, of the corpus of images. That is, multiple characteristics, not just one, are used to determine whether a user is paying attention to a computing screen or not.
[0022] With multiple user position characteristics extracted for a stream of images for a particular user and user position characteristics extracted for a reference corpus of images, a comparer (106) determines, by comparing the stream of images with the images in the database, whether the user is paying attention to the computing device. That is, the comparer (106) can determine whether the stream of images includes user position characteristics more consistent with images indicative of paying attention or with images indicative of a lack of attention. Based on such a comparison, a determination is made regarding user attention, and a notification may be presented to a user. For example, a notice may be provided to a user of the computing device reminding them to return their attention to the computing device. A notice may also be provided to an administrator of the environment such that remedial action may be taken. For example, in the case of a classroom setting, an output of each student’s level of attention may be provided to an instructor such that the instructor may take remedial action to bring the student’s attention back to the computing devices.
[0023] Thus, in an education setting, an instructor may be provided with a management console that can gauge the state of the classroom based on its attention levels. On an individual level, the instructor is notified that a student may be having a tough time focusing. This can give the instructor an opportunity to ask the student if they need help, to find a new assignment to re- interest the student, or to help the instructor control a situation.
[0024] On a group level, if multiple students’ attention are compromised, then the instructor can be notified that the classroom is becoming less focused. Responsive to such a determination, the instructor may change activities or move on to another subject. This information would be especially useful for remote instructors, who cannot be there in full capacity. Additionally, this same protocol can be used to indicate whether or not a student might be cheating on an assignment by looking over at another student’s computer. Quick
notifications to the instructor management console can alert the instructor quickly such that they may assess the situation.
[0025] In some examples, the stream of images/video of a user is collected from an integrated camera on a user’s computing device. For example, a laptop computer or desktop may include a camera that can capture the images from which user position characteristics are extracted. In another example, a separate camera, such as one in the classroom may be used to determine student attention.
[0026] Accordingly, such an attention analysis system (100) provides a simply way to determine whether a single user, or multiple users, are focusing on a computing device screen. This may include analyzing more information than just eye gaze. That is other user position information such as head position, body position, and eye position may be relied on. Relying on more than just eye gaze provides a more accurate determination as to attention. That is, eye gaze alone is a single data point and may not accurately describe whether a user is paying attention or not. In a specific example, a user’s eyes may be directed towards a computer screen. However, the user’s eyelids may be partially closed indicating a user is not paying attention. Accordingly, the system (100) by relying on multiple user position characteristics provides a more accurate determination of user attention to a computing device screen.
[0027] Fig. 2 is a flow chart of a method (200) for computing device attention determination, according to an example of the principles described herein. According to the method (200), a stream of images is collected (block 201 ) of a user in front of a computing device. As described above, the stream of images may be still images or may be images strung together as frames of a video clip. The images may be collected (block 201 ) from a camera integrated into the computing device of the user or may be collected from another source such as a camera(s) disposed in the room with the user. In either case, the camera captures images of the user as they are sitting in front of a respective computing device.
[0028] Multiple user position characteristics are extracted (block 202) from the stream of images. Such user position characteristics include eye gaze and additional characteristics. For example, user eye information may include additional information relating to a position/movement of a user’s eye. As specific examples, user eye information may include a gaze direction, pupil dilation, eye movement, etc. Additional pieces of eye position information provide better results regarding user attention. That is, while an eye gaze towards a user computing device is indicia of user attention, pupil dilation may be also be indicative of user attention. As yet another example eye movement may be indicative of user attention. The combination of this information provides more information than eye gaze alone regarding user attention.
[0029] For example, it may be the case that a user is not looking at a computing device screen, but is looking down at a keyboard. Eye gaze alone may indicate that this user is not paying attention. However, knowing that the computing device includes a keyboard, eye movement information may indicate that a user’s eyes are frequently going from a downward position (looking at the keyboard) to an upward position (looking at the screen). Thus, even though a user’s gaze is not entirely at the computing device, they may still be paying attention, albeit with their gaze in a different direction, i.e., at the keyboard or looking at a text book.
[0030] The attention analysis system (Fig. 1 , 100) may extract (block 202) additional characteristics, in addition to eye position information. For example, user head position information that indicates a head position/movement may also indicate user attention. Examples of such head position/movement may indicate that a user is not looking at the computing device screen. For example, a head position angled away from the computing device screen indicates a user is not paying attention. Head position information may also be used to expound upon information collected eye position information. For example, a user may rest their head on their hand at an angle while looking at the computing device screen. Eye gaze alone may therefore indicate a user is paying attention, as their eyes are in the direction of the computing screen. However, the resting of the head on the propped hand of a user is indicia that a user, despite looking at the screen, is not paying attention.
[0031] Other examples of head position information include head feature position information, such as a position/movement of eyelids, eyebrows, nose, and/or mouth. As a specific example droopy eyelids may indicate that again, while a user gaze may be towards a computing device screen, a user is in fact not paying attention. Thus, the present specification by relying on user position information beyond eye gaze provides more details and more data points such that a more accurate determination of whether a user is paying attention or not may be made.
[0032] In yet another example, the information may include user body position information. For example, shoulder angle information may be indicative of whether a user is slouching in their chair, which may be indicia that a user is not paying attention. While specific reference is made to particular body position characteristics of a user, other body position/movement characteristics may be extracted (block 202) from a stream of images for a user by which it may be determined whether that user is paying attention or not.
[0033] The stream of images, and more particularly the extracted characteristics from the stream of images, are compared (block 203) against a database (Fig. 1 , 102) of images. The database (Fig. 1 , 102) of images against which the stream of images are compared have been similarly analyzed with characteristics extracted therefrom. As described, the database (Fig. 1 , 102) of images includes a training set which includes images and a classification as to whether it indicates a user is paying attention or not. The features of these images are analyzed such that user position characteristics may be mapped to either an attentive or non-attentive user. [0034] Thus, the comparison (block 203) determines whether the stream of images shares more common characteristics with images reflecting high attention or with images reflecting poor attention. In some examples, the comparison (block 203) of the stream of images may be performed over a period of time. For example, it may not be the case that a user looking away from his/her computing device screen for a few seconds indicates they are not paying attention. However, the user looking away from his/her computing device for 10-15 seconds may be more indicative of a lack of attention. That is, distraction is gauged in combination with the length of time that a user’s face is positioned away from the screen and the number of occurrences. Outliers and anomalies are considered momentary pauses and rests. Recurring infractions by comparison, provide a more sound indication of user distraction.
Accordingly, when the image gathered in real-time execution triggers a negative state, that action is logged and tracked over time.
[0035] With such information extracted (block 202) and compared (block
203) over time, the attention analysis system (Fig. 1 , 100) determines (block
204), based on the comparison, whether the user is paying attention to the computing device. Such a determination (block 204) may be made based on a degree of similarity between the stream of images and images of the different subsets in the database (Fig. 1 , 102). That is, the attention analysis system (Fig. 1 , 100) determines with which of the subset of images the stream of images has more in common. That is, whether the stream of images has more in common with the images indicative that a user is paying attention to a corresponding computing device or whether the stream of images has more in common with the images indicative that a user is not paying attention to a corresponding computing device. As used in the present specification and in the appended claims the term“in common” refers to a quantity of shared user position characteristics of the images. Note that such a determination is not based on a single factor, i.e. , eye gaze, but whether the image on the whole shares more common characteristics (e.g., eyelid position, mouth movement, head angle, appendage position, body angle), with one or the other set of images. [0036] Thus, the present specification describes a method (200) that relies on a reliable set of images, i.e. the training set which may include thousands or even millions of examples, from which characteristics indicative of attention or distraction are drawn. Comparing a set of images depicting a particular user against these images allows for a determination as to whether a user is paying attention to their computing device.
[0037] Fig. 3 is a block diagram of a system (100) for computing device attention determination, according to another example of the principles described herein. As described above in connection with Fig. 1 , the attention analysis system (100) includes a database (102), an analyzer (104), and a comparer (106). In some examples, the attention analysis system (100) includes additional components.
[0038] For example, the attention analysis system (100) includes an aggregator (308) to generate an output resulting from the analysis of different sets of images, each set corresponding to a different user. That is, the aggregator (308) generates an output based on comparisons of multiple streams of images relating to multiple users with the images in the database (102). In so doing, an aggregate level of attention of multiple users can be indicated. That is, a different set of remedial actions may be carried out if multiple users are not paying attention as opposed to a single user not paying attention.
[0039] For example, in an educational setting if multiple users are not paying attention, it may be indicative that the current pedagogical technique is not effective, or that an instructor has spent too much time on a particular topic and that the students in the classroom could benefit from a change in subject. The information collected by the aggregator (308) may also be used to evaluate a presenter, teacher, or administrator. For example, if a majority of the audience is not paying attention to a particular presentation, as indicated by a comparison of the stream of images for those users against the database of images, it may indicate that the presenter, teacher, or administrator has not effectively presented the material. By comparison, if a majority of users are actively engaged in the presentation, it may indicate that the presenter, teacher, or administrator has been effective in delivery the content.
[0040] Still further, in an example, the attention analysis system (100) includes an identifier (310) to identify patterns in the aggregate level of attention. For example, it may be the case that a certain group of users are not paying attention, but that others are. The identifier (310) may identify such patterns. As a specific example, it may be the case that a particular region within the classroom has students that are not paying attention. This may be presented on the management console of the instructor as a heat map, thus suggesting that perhaps students in that area should be separated from one another.
[0041] In yet another example, the identifier (310) may indicate that users within a certain demographic, for example a certain age range, are more prone to falling out of attention as compared to another age range. Accordingly, this may be presented to the administrator to tailor the subject matter to be more wide-reaching in regards to audience demographics.
[0042] Accordingly, the additional components within the attention analysis system (100) provide additional information that can be used to improve user focus. In some examples, the information collected could be transmitted elsewhere. For example, as described above an aggregate level of attention may be passed such that an evaluation of the presenter or instructor may be made. In another example, the aggregate level of attention as well as the output of the identifier (310) may be passed to enhance and alter the subject matter, or form, of a presentation. For example, it may be the case that a video presentation shown in a dark room leads to many users falling asleep as indicated by a comparison of body position features of the various users compared against images in the database. Accordingly, the presentation of the material may be changed to an interactive multimedia presentation rather than a video display.
[0043] Fig. 4 is a flow chart of a method (400) for computing device attention determination, according to another example of the principles described herein. According to the method (400), a stream of images of a user in front of a computing device is collected (block 401 ) and user position characteristics are extracted (block 402) therefrom. This stream of images, and more particularly the extracted user position characteristics, are compared (block 403) against images in a training database. These operations may be carried out as described above in connection with Fig. 2.
[0044] In addition to user position information, the attention analysis system (Fig. 1 , 100) may rely on other information as well. That is, while user position information may provide indicia of user attention, additional information may provide a more accurate determination. For example, a user may be looking at a computing device, while the user’s mind may be elsewhere. Thus, eye gaze information alone may indicate a user is paying attention, however additional criteria may indicate that the user is not paying attention to the computer screen that they are looking at.
[0045] Accordingly, the attention analysis system (Fig. 1 , 100) may rely on other pieces of information, specifically user interaction with the computing device. For example, user interaction with the computing device may include a user manipulation of a user input device. For example, a user may move a mouse cursor during an interactive display, or may enter information via a keyboard. This interaction may indicate attention to the presentation, regardless of an eye gaze direction. While particular reference is made to a few specific user input devices, multiple other types of user input devices may exist and a user’s interaction with those devices may be collected (block 404).
[0046] In another example, user interaction may be indicated by a change of information displayed on the computing device. For example, a multimedia presentation may be displayed on the computing device. In this example, the attention analysis system (Fig. 1 , 100) may detect a change in display content for example as a user advances through slides of the presentation. Such a change in displayed content can be considered in addition to the comparison (block 403) of images with the database to determine whether or not the user is actively paying attention to the computing device.
[0047] In another example, noise level information around the computing device may be collected (block 405). That is, ambient noise in a space can be a source of distraction to individuals. Accordingly, a microphone on a computing device or in a room may be used to determine a level of ambient noise around the user. This information, in addition to user position characteristics and user interaction information can provide a more accurate view of user focus.
[0048] Accordingly, in determining (block 406) whether a user is paying attention, all pieces of information may be considered. That is, the results of the comparison (block 403), the collected (block 404) information regarding user interaction with the computing device, and the collected (block 405) information regarding noise levels around the computing device all may be used in conjunction with one another by the attention analysis system (Fig. 1 , 100) to determine user attention. As such, the present method (400) provides an environment where multiple pieces of different types of information, not just related to the user movement/position information, are used in determining user attention. Thus, the attention analysis system (Fig. 1 , 100) and the computing device in which it is installed can provide a more accurate picture regarding user attention such that more targeted remedial measures, if needed, may be executed.
[0049] Fig. 5 is a diagram of an environment for computing device (512) attention determination, according to an example of the principles described herein. As described above, the attention analysis system (100) is coupled to a network of computing devices (512). One of those computing devices (512-1 ) is associated with a user of whom it is desired to know if he/she is paying attention to the computing device (512-1 ). The other computing devices (512-2, 512-3, 512-4, 512-5) provide the training set of images that are included in the database (Fig. 1 , 102) to which the stream of images from the computing device (512-1 ) are compared.
[0050] As described above, this information is collected and analyzed by the attention analysis system (100) to determine a first set of characteristics that map to images of users paying attention to their computing device screens and a second set of characteristics that map to images of users not paying attention to their computing device screens. [0051] The attention analysis system (100) also collects streams of images (514) from the first computing device (512-1 ). As described above, the images (514) may be still images taken over a period of time. In another example, the images (514) may be frames of a video clip also taken over a period of time.
[0052] The computing device (512) also transmits other information (516) such as information indicating a user interaction with the computing device (512- 1 ) and noise levels around the computing device (512-1 ). All this information is processed by the attention analysis system (100) to determine whether a user is paying attention. Such an analysis may be performed for any number of computing devices (512). That is, such an analysis may be done for a single computing device (512) to ascertain the attention level of one user or such analysis may be done for multiple computing devices (512) to ascertain the attention level of multiple users. The information is then output by the attention analysis system (100). That is, the information may be output via a
management console wherein a user of the management console can address individual distraction or aggregate distraction. Moreover, in some examples the information is transmitted further, for example to a server, where it is
subsequently used for further analysis such as content development and/or instructor evaluation.
[0053] Fig. 6 is a diagram of a machine-readable storage medium (618) for computing device attention determination, according to an example of the principles described herein. To achieve its desired functionality, a computing system includes various hardware components. Specifically, the computing system includes a processor. Machine-readable storage medium (618) is communicatively coupled to the processor. The machine-readable storage medium (618) includes a number of instruction sets (620, 622, 624, 626, 628, 630, 632) for performing a designated function. The machine-readable storage medium (618) causes the processor to execute the designated function of the instruction sets (620, 622, 624, 626, 628, 630, 632).
[0054] Although the following descriptions refer to a single machine- readable storage medium (618), the descriptions may also apply to multiple machine-readable storage mediums. In such examples, the instruction sets (620, 622, 624, 626, 628, 630, 632) may be distributed (e.g., stored) across multiple machine-readable storage mediums.
[0055] The machine-readable storage medium (618) represents any tangible and non-transitory memory capable of storing data such as
programmed instructions or data structures used by the computing system.
[0056] Referring to Fig. 6, corpus instructions (620), when executed by a processor, may cause the computing system to, collect from multiple computing devices (Fig. 5, 512) a corpus of images that include images indicative that users are paying attention to corresponding computing devices and images indicative that users are not paying attention to corresponding computing devices. Collect instructions (622), when executed by a processor, may cause the computing system to collect from a computing device (Fig. 5, 512-1 ), a stream of images (Fig. 5, 514) of a user disposed in front of the computing device. Extract instructions (624), when executed by a processor, may cause the computing system to extract multiple user position characteristics from the stream of images (Fig. 5, 514). Compare instructions (626), when executed by a processor, may cause the computing system to compare the stream of images (Fig. 5, 514) to the corpus of images. Interaction instructions (628), when executed by a processor, may cause the computing system to collect information regarding a user interaction with the computing device (Fig. 5, 512-1 ). Determine instructions (630), when executed by a processor, may cause the computing system to, based on a comparison of the stream of images (Fig. 5, 514) to the corpus of images and the information regarding the user interaction with the computing device (Fig. 5, 512-1 ), determine whether the user is paying attention to the computing device (Fig. 5, 512-1 ).
Notice instructions (632), when executed by a processor, may cause the computing system to provide notice regarding whether the user is paying attention to the computing device (Fig. 5, 512-1).
[0057] In summary, using such an attention determining system 1 ) allows for a systematic and effective way to track focus to material being presented on a computing device; 2) allows remote presenters information regarding audience engagement; and 3) allows presenters to focus on preparation and presentation of material without worrying about audience distraction.

Claims

CLAIMS What is claimed is:
1. An attention determining system, comprising:
a database comprising:
images indicative that users are paying attention to corresponding computing devices; and
images indicative that users are not paying attention to corresponding computing devices;
a machine-learning analyzer to extract multiple user position
characteristics from a stream of images of a user disposed in front of a computing device; and
a comparer to determine, by comparing the stream of images of the user to the images in the database, whether the user is paying attention to the computing device.
2. The system of claim 1 , further comprising an aggregator to generate an output, based on comparisons of multiple streams of images relating to multiple users with the images in the database, indicative of an aggregate level of attention of the multiple users.
3. The system of claim 2, further comprising an identifier to identify patterns in the aggregate level of attention.
4. The system of claim 1 , wherein the user is a student in a classroom and an output of the comparer for the student is provided to an instructor.
5. A method, comprising:
collecting from a computing device, a stream of images of a user disposed in front of the computing device; extracting multiple user position characteristics from the stream of images;
comparing the stream of images to a database comprising:
images indicative that users are paying attention to corresponding computing devices; and
images indicative that users are not paying attention to corresponding computing devices; and
determining, based on the comparison, whether the user is paying attention to the computing device.
6. The method of claim 5, wherein the user position characteristics comprise at least one of:
user eye position information;
user head position information; and
user body position information.
7. The method of claim 5:
further comprising collecting information regarding a user interaction with the computing device; and
wherein determining whether the user is paying attention to the computing device is further based on collected information regarding the user interaction with the computing device.
8. The method of claim 5, wherein comparing the stream of images to the database is performed over a period of time.
9. The method of claim 5:
further comprising collecting information regarding a noise level around the computing device; and
wherein determining whether the user is paying attention to the computing device is further based on the noise level around the computing device.
10. The method of claim 5, wherein determining, based on the comparison, whether the user is paying attention to the computing device comprises determining with which of 1 ) the images indicative that users are paying attention to corresponding computing devices and 2) the images indicative that users are not paying attention to corresponding computing devices, the stream of images has more shared characteristics.
1 1. A tangible machine-readable storage medium encoded with instructions executable by a processor, the machine-readable storage medium comprising: instructions to collect from multiple computing devices, a corpus of images comprising:
images indicative that users are paying attention to corresponding computing devices; and
images indicative that users are not paying attention to corresponding computing devices;
instructions to collect from a computing device, a stream of images of a user disposed in front of the computing device;
instructions to extract multiple user position characteristics from the stream of images;
instructions to compare the stream of images to the corpus of images; instructions to collect information regarding a user interaction with the computing device;
instructions to, based on a comparison of the stream of images to the corpus of images and the information regarding the user interaction with the computing device, determine whether the user is paying attention to the computing device; and
instructions to provide notice regarding whether the user is paying attention to the computing device.
12. The tangible machine-readable storage medium of claim 1 1 , wherein the user interaction with the computing device comprises interaction with a user input device of the computing device.
13. The tangible machine-readable storage medium of claim 1 1 , wherein the user interaction with the computing device comprises a change in information displayed on a display of the computing device.
14. The tangible machine-readable storage medium of claim 1 1 , further comprising instructions to transmit results of the determination of whether the user is paying attention to the computing device to a server.
15. The tangible machine-readable storage medium of claim 1 1 , wherein notice is provided to at least one of:
a user of the computing device; or
an administrator of the computing device.
PCT/US2018/048880 2018-08-30 2018-08-30 Computing device attention determination WO2020046346A1 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP18932010.4A EP3759650A4 (en) 2018-08-30 2018-08-30 Computing device attention determination
CN201880092960.1A CN112005248A (en) 2018-08-30 2018-08-30 Computing device attention determination
US17/049,193 US12002287B2 (en) 2018-08-30 2018-08-30 Computing device attention determination
PCT/US2018/048880 WO2020046346A1 (en) 2018-08-30 2018-08-30 Computing device attention determination

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2018/048880 WO2020046346A1 (en) 2018-08-30 2018-08-30 Computing device attention determination

Publications (1)

Publication Number Publication Date
WO2020046346A1 true WO2020046346A1 (en) 2020-03-05

Family

ID=69644558

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2018/048880 WO2020046346A1 (en) 2018-08-30 2018-08-30 Computing device attention determination

Country Status (4)

Country Link
US (1) US12002287B2 (en)
EP (1) EP3759650A4 (en)
CN (1) CN112005248A (en)
WO (1) WO2020046346A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12002287B2 (en) 2018-08-30 2024-06-04 Hewlett-Packard Development Company, L.P. Computing device attention determination

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220192560A1 (en) * 2020-12-23 2022-06-23 Qatar Foundation For Education, Science And Community Development Systems and methods for evaluating mindfulness

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6246779B1 (en) * 1997-12-12 2001-06-12 Kabushiki Kaisha Toshiba Gaze position detection apparatus and method
US20110279676A1 (en) * 2009-10-15 2011-11-17 Panasonic Corporation Driving attention amount determination device, method, and computer program
US20140026156A1 (en) * 2012-07-18 2014-01-23 David Deephanphongs Determining User Interest Through Detected Physical Indicia
US20140121002A1 (en) * 2005-09-15 2014-05-01 Sony Computer Entertainment Inc. System and method for detecting user attention
US20160217321A1 (en) 2015-01-23 2016-07-28 Shindig. Inc. Systems and methods for analyzing facial expressions within an online classroom to gauge participant attentiveness
US20180077455A1 (en) * 2016-09-14 2018-03-15 International Business Machines Corporation Attentiveness-based video presentation management

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5944530A (en) 1996-08-13 1999-08-31 Ho; Chi Fai Learning method and system that consider a student's concentration level
JP4594945B2 (en) * 2007-02-13 2010-12-08 株式会社東芝 Person search device and person search method
CN102799893B (en) 2012-06-15 2015-04-22 北京理工大学 Method for processing monitoring video in examination room
US8847771B2 (en) * 2013-01-25 2014-09-30 Toyota Motor Engineering & Manufacturing North America, Inc. Method and apparatus for early detection of dynamic attentive states for providing an inattentive warning
US9041766B1 (en) * 2013-03-14 2015-05-26 Ca, Inc. Automated attention detection
US9892315B2 (en) 2013-05-10 2018-02-13 Sension, Inc. Systems and methods for detection of behavior correlated with outside distractions in examinations
CN104517102B (en) 2014-12-26 2017-09-29 华中师范大学 Student classroom notice detection method and system
US10303245B2 (en) * 2015-05-04 2019-05-28 Adobe Inc. Methods and devices for detecting and responding to changes in eye conditions during presentation of video on electronic devices
CN106778676B (en) 2016-12-31 2020-02-18 中南大学 Attention assessment method based on face recognition and image processing
CN106952200A (en) 2017-03-28 2017-07-14 安徽味唯网络科技有限公司 A kind of method that internet teaching supervises student
CN107154202A (en) 2017-07-12 2017-09-12 昆明理工大学 Smart classroom system
US12002287B2 (en) 2018-08-30 2024-06-04 Hewlett-Packard Development Company, L.P. Computing device attention determination

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6246779B1 (en) * 1997-12-12 2001-06-12 Kabushiki Kaisha Toshiba Gaze position detection apparatus and method
US20140121002A1 (en) * 2005-09-15 2014-05-01 Sony Computer Entertainment Inc. System and method for detecting user attention
US20110279676A1 (en) * 2009-10-15 2011-11-17 Panasonic Corporation Driving attention amount determination device, method, and computer program
US20140026156A1 (en) * 2012-07-18 2014-01-23 David Deephanphongs Determining User Interest Through Detected Physical Indicia
US20160217321A1 (en) 2015-01-23 2016-07-28 Shindig. Inc. Systems and methods for analyzing facial expressions within an online classroom to gauge participant attentiveness
US20180077455A1 (en) * 2016-09-14 2018-03-15 International Business Machines Corporation Attentiveness-based video presentation management

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3759650A4

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12002287B2 (en) 2018-08-30 2024-06-04 Hewlett-Packard Development Company, L.P. Computing device attention determination

Also Published As

Publication number Publication date
US12002287B2 (en) 2024-06-04
EP3759650A4 (en) 2021-10-27
EP3759650A1 (en) 2021-01-06
CN112005248A (en) 2020-11-27
US20210240968A1 (en) 2021-08-05

Similar Documents

Publication Publication Date Title
Bosch et al. Automatic detection of learning-centered affective states in the wild
US20220319181A1 (en) Artificial intelligence (ai)-based system and method for managing education of students in real-time
Li et al. Massive open online proctor: Protecting the credibility of MOOCs certificates
US7760910B2 (en) Evaluation of visual stimuli using existing viewing data
Greene et al. Under high perceptual load, observers look but do not see
WO2017018012A1 (en) Information processing system, information processing method, and storage medium
CN116018789A (en) Method, system and medium for context-based assessment of student attention in online learning
Kar et al. Gestatten: Estimation of User's attention in Mobile MOOCs from eye gaze and gaze gesture tracking
US12002287B2 (en) Computing device attention determination
Sheth et al. Detecting social and non-social changes in natural scenes: Performance of children with and without autism spectrum disorders and typical adults
WO2023041940A1 (en) Gaze-based behavioural monitoring system
Prieto et al. Studying teacher cognitive load in multi-tabletop classrooms using mobile eye-tracking
US20240169855A1 (en) Systems and methods for assessing data collected by automated proctoring
Byrne et al. Leveraging eye tracking in digital classrooms: A step towards multimodal model for learning assistance
Dacayan et al. Computer-Vision Based Attention Monitoring for Online Meetings
Anisimov et al. OkenReader: ML-based classification of the reading patterns using an Apple iPad
WO2022168180A1 (en) Video session evaluation terminal, video session evaluation system, and video session evaluation program
WO2022168185A1 (en) Video session evaluation terminal, video session evaluation system, and video session evaluation program
US20230127335A1 (en) Intelligent and adaptive measurement system for remote education
KR20230030683A (en) Platform for online learning concentration improvement and method thereof
Lyu et al. Optical character recognition (OCR)-based and Gaussian mixture modeling-OCR-based slide-level “with-me-ness”: Automated measurement and feedback of learners’ attention state during video lectures
Castillon et al. Automatically Identifying the Human Sense of Familiarity Using Eye Gaze Features
WO2022168174A1 (en) Video session evaluation terminal, video session evaluation system, and video session evaluation program
WO2022168178A1 (en) Video session evaluation terminal, video session evaluation system, and video session evaluation program
WO2022168179A1 (en) Video session evaluation terminal, video session evaluation system, and video session evaluation program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18932010

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2018932010

Country of ref document: EP

Effective date: 20200929

NENP Non-entry into the national phase

Ref country code: DE