WO2021262166A1

WO2021262166A1 - Operator evaluation and vehicle control based on eyewear data

Info

Publication number: WO2021262166A1
Application number: PCT/US2020/039492
Authority: WO
Inventors: Subrata Kumar KUNDU
Original assignee: Hitachi America, Ltd.
Priority date: 2020-06-25
Filing date: 2020-06-25
Publication date: 2021-12-30

Abstract

In some examples, a computing device may receive, from eyewear including one or more cameras, at least one image. For example, the eyewear may be worn by an operator of a vehicle. The computing device may determine, based at least in part on the at least one image, an operator field of view (FOV). In addition, the computing device may compare the operator FOV with a recommended FOV. Further, the computing device may perform at least one action based on comparing the operator FOV with the recommended FOV.

Description

OPERATOR EVALUATION AND VEHICLE CONTROL BASED ON

EYEWEAR DATA

TECHNICAL FIELD

[0001] This disclosure relates to the technical fields of monitoring a vehicle operator and autonomous or semiautonomous control of a vehicle.

BACKGROUND

[0002] With increasing global economic growth and urbanization, roads are becoming busier, which has increased the focus on the safety of vehicles as well as safe operation by vehicle operators. Ensuring safe driving not only applies to individuals but also may be a priority for any type of transportation business. Recent trends indicate that companies with fleets of vehicles consider safety and safe driving behavior as one of their top priorities. Monitoring driving behavior not only increases the safety of vehicle occupants, but good driving behavior may also help to reduce fuel and maintenance costs, reduce vehicle downtimes, and avoid freight damage. [0003] To realize improved vehicle safety, autonomous driving (AD) and advanced driver assistance systems (ADAS) are rapidly being adopted by the automotive industry. ADAS and AD systems may employ many different types of sensors such as mono cameras, stereo cameras, infrared cameras, radar, lidar, GPS receivers, ultrasonic sensors, etc., to capture and integrate various different types of information. For instance, information regarding the environment around the vehicle captured by these sensors may be used as input information to a computer or other controller that may intelligently control a vehicle or various systems of the vehicle, such as for avoiding collisions, reducing fuel consumption, and increasing the convenience and safety of the vehicle occupants.

SUMMARY

[0004] Some implementations include a computing device that may receive, from eyewear including one or more cameras, at least one image. For example, the eyewear may be worn by an operator of a vehicle. The computing device may determine, based at least in part on the at least one image, an operator field of view (FOV). In addition, the computing device may compare the operator FOV with a recommended FOV. Further, the computing device may perform at least one action based on comparing the operator FOV with the recommended FOV. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The detailed description is set forth with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical items or features.

[0006] FIG. 1 illustrates an example architecture of a system able to evaluate an operator and/or perform vehicle control based on eyewear data according to some implementations.

[0007] FIG. 2 is a flow diagram illustrating an example process for monitoring an operator and determining an action according to some implementations.

[0008] FIG. 3A illustrates an example of feature estimation according to some implementations .

[0009] FIG. 3B illustrates examples of eye gaze estimation according to some implementations .

[0010] FIG. 4 illustrates an example of a membership function for determining horizontal eye gaze according to some implementations.

[0011] FIG. 5 illustrates an example of a head pose coordinate system according to some implementations .

[0012] FIG. 6 illustrates an example of the calibration image according to some implementations .

[0013] FIG. 7A illustrates an example of detecting distraction by detecting that the operator is looking out the side window according to some implementations.

[0014] FIG. 7B illustrates an example of detecting distraction according to some implementations .

[0015] FIG. 8 illustrates an example membership function for distraction variables according to some implementations.

[0016] FIG. 9A is a flow diagram illustrating the example process for determining a heart rate of an operator according to some implementations.

[0017] FIG. 9B is a flow diagram illustrating the example process for determining a fatigue level of an operator according to some implementations.

[0018] FIG. 10 illustrates an example architecture of a system and operations performed by the system according to some implementations.

[0019] FIG. 11 is a flow diagram illustrating an example process for monitoring an operator according to some implementations. [0020] FIG. 12 is a flow diagram illustrating an example process for monitoring an operator and/or controlling a vehicle according to some implementations.

DESCRIPTION OF THE EMBODIMENTS

[0021] Some implementations herein are directed to techniques and arrangements for monitoring, evaluating and rating a vehicle operator. Further, some implementations are directed to autonomous vehicle control techniques that may utilize data from smart eyewear such as smart eyeglasses and/or smart contact lenses that may be worn by a vehicle operator. For example, the system herein may use image data or other information from the eyewear to continuously monitor the physiological condition of a vehicle operator and provide feedback to the operator and/or provide automated vehicle control accordingly. In some examples, a vehicle ECU (electronic control unit), such as an ADAS/AD ECU or other vehicle computing device may perform at least a portion of the automated operations herein. In other examples, a mobile device, such as a smart phone, tablet, or the eyewear itself may perform a portion of the processing operations described herein. As one example, a mobile device may perform image processing on images received from the eyewear, while an ECU or other vehicle computing device may receive information about the processed images and may perform vehicle control operations and/or may provide notifications to the vehicle operator. As another example, one or more vehicle computing devices may perform all the operations and a mobile device is not employed for performing the data processing/decision making.

[0022] In some cases, the system herein may receive images as input information from the smart eyewear through wireless communication with the vehicle operator’s smart glasses/lenses. For example, the images obtained from the smart eyewear may include a representation of what the wearer actually sees, also referred to herein as the operator’s field of view (FOV). In addition, in some examples, the received images received from the smart eyewear may include images of operator eyes. The system herein may process the images and determine the operator’s driving performance and current physiological condition. The system may store the results in a local storage and/or may upload or otherwise send the determined information to a cloud database or other service computing device that may be able to communicate with the system via a network. Further, the system may provide a recommendation, suggestion, alert, message or other type of notification to the vehicle operator, such as by causing the notification to be presented on a display of the vehicle and/or a display, projector, etc., of the smart eyewear itself. In addition, the system may send one or more control signals to one or more vehicle control ECUs for directly controlling the vehicle or one or more vehicle systems, such as brakes, steering, parking, acceleration, or the like.

[0023] In case the system detects an issue with the operator, such as drowsiness, fatigue, distraction, and so forth, the system may send a notification to a display unit of the smart glasses/lenses. For example, the smart eyewear may project the notification image directly into the eye of the operator without hindering the operator’s normal vision. The system herein may also provide a recommended required field of view (FOV) area corresponding to the current location of the vehicle. For example, if there is a turn coming up, the recommended FOV area may include the lane in which the vehicle will be turning. The recommended FOV area may be displayed as an augmented reality holographic image so that operator can pay attention to the specific area to ensure that the turn is navigated safely.

[0024] Implementations herein are able to monitor a vehicle operator’ s driving performance and physiological condition which may help to significantly improve safety. Wearable eyewear device data may be used to monitor vehicle operator attentiveness and other conditions for providing notifications to the operator as well as operate the vehicle safely in case of an emergency. Accordingly, the examples herein may provide significant benefits for both fully autonomous and semi-autonomous driving vehicles that are able to accommodate human operators. The system herein may be executed using a vehicle computing device, as well as a mobile device, such as smartphones, tablets, wearable devices, etc. Furthermore, implementations herein are not limited to use with passenger automobiles, but may also be applied to monitoring operators of various other types of vehicles such as shuttle buses, trains, boats, ships, and so forth, utilizing images from the smart eyewear devices.

[0025] In some examples, the operator monitoring system herein may be used as a kind of ADAS application that may be employed to evaluate and rate an operator. For example, the eyewear device may include a camera that may detect the operator’ s eyes to continuously monitor the operator for detecting a focal location or other eye gaze of the operator. In addition, the camera or other sensors in the eyewear may be used to check for conditions such as drowsiness, distraction, emotion, health conditions (e.g., irregular heartbeat), or the like. In addition, based on the operator evaluation and rating, the results determined by the system may be can be used to warn the operator and/or to control the vehicle to ensure safety. As one example, the system output may be used to activate a lane keep assistance (LKA) ADAS application when drowsiness has been detected.

[0026] Some examples herein may also utilize vehicle information such as vehicle speed (to compare with allowable speed), vehicle acceleration and deceleration, steering angle, stopping behavior at traffic lights and stop signs, etc. Based on this information, a driving behavior evaluation and rating may be performed. Smart eyewear devices be configured to perform a number of useful functions such as capturing capture images of what the wearer sees, recording the images, receiving and sending messages and phone calls, providing for navigation functions and augmented reality, enabling interaction with social media and applications, managing calendars, appointments, and so forth. In some examples, the smart glasses herein are able to project images at a certain distance from the eyes of the wearer to display various different types of information to the wearer. For instance, the smart glasses may include a projector that shines an image on a holographic mirror that bounces the image into the wearer’ s eye(s) thereby enabling information to be displayed to the wearer without interfering with the wearer’ s normal vision. In addition, the smart glasses and smart contact lenses may be configured to capture images of what the wearer sees. For instance, both smart glasses and smart contact lenses may be equipped with a camera, processor, data storage, display capability, battery, and so forth, and may be able to communicate wirelessly such as by using BLUETOOTH®, Wi-Fi, cellular radio, etc.

[0027] Implementations herein use images from smart eyewear worn by vehicle operators to rate the vehicle operators, evaluate the operators’ psychophysiological conditions and attention levels, and monitor consecutive actions to control a conventional manual or semi-autonomous vehicle. Furthermore, the system may send information related to the vehicle operation and/or the operator’s current condition to the eyewear for presentation to the operator. The system may also provide the information over a network to a server, or the like, such as for maintaining a record of the physiological condition of the operator, operator performance, operator rating, and the like. Accordingly, in some examples herein, the system estimates an operator’ s physiological condition and rates the operator’s driving performance using only smart glasses or/and smart lenses data. Thus, in some examples, the system herein does not require any dashboard or cabin mounted camera, or other physiological or biometric sensors in other areas of the vehicle. Alternatively, in other examples, such as in the case of smart contact lenses, in which the smart eyewear does not include a camera able to capture an image of the operator’s eye(s), a driver- facing cabin- mounted camera (such as a dashboard camera, visor camera, or the like) may be used to perform some of the described functions. In these examples, the images from the cabin-mounted camera may be processed in addition to, and concurrently with, the images and/or other data from the smart eyewear.

[0028] For discussion purposes, some implementations are described in the monitoring a vehicle operator and operation of a vehicle based on data from smart eyewear worn by the operator. However, implementations herein are not limited to the particular examples provided, and may be extended to other types of eyewear devices, other types of vehicles, other types of data, other types of sensors, and so forth, as will be apparent to those of skill in the art in light of the disclosure herein.

[0029] FIG. 1 illustrates an example architecture of a system 100 able to evaluate an operator and/or perform vehicle control based on eyewear data according to some implementations. The system 100 includes a vehicle 102 having one or more vehicle computing devices 104. The vehicle computing device(s) 104 may be able to communicate wirelessly with smart eyewear 106, such as smart glasses 108 and/or smart contact lenses 110. In some cases, the vehicle computing device 104 may communicate directly with the smart eyewear 106 while in other cases, the communication may be relayed through a mobile device 112 of a vehicle operator 114. For instance, the vehicle operator 114 may wear the smart glasses 108 and/or the smart contact lenses 110 while operating the vehicle 102.

[0030] Each vehicle computing device 104 may include one or more processors 116, one or more computer-readable media 118, and one or more communication interfaces (I/Fs) 120. In some examples, the vehicle computing device(s) 104 may include one or more ECUs (electronic control units) or any of various other types of computing devices. For instance, the computing device(s) 104 may include one or more ADAS/AD ECUs which may execute one or more programs for controlling at least some vehicle systems 122, such as to perform ADAS and/or AD tasks, which may include navigation, braking, steering, acceleration, deceleration, and so forth. The computing device(s) 104 may also include one or more other ECUs, such as for controlling other systems of the vehicle systems 122.

[0031] “ECU” may include any embedded processing system that controls one or more of the systems, subsystems, or components in a vehicle. Software, such as a control program 124 and an image processing program 126 may be executed by one or more ECUs and may be stored in a portion of the computer-readable media 118 (e.g., program ROM, solid state storage, etc., as discussed below) associated with the respective ECU to enable the ECU to operate as an embedded system. ECUs may typically communicate with each other over a vehicle bus, such as a controller area network (CAN) bus according to a vehicle bus protocol. As an example, the CAN bus protocol is a vehicle bus protocol that allows ECUs and the vehicle systems 122 to communicate with each other without a host computer.

[0032] Each ECU or other vehicle computing device 104 may include one or more processors 116, which may include one or more of central processing units (CPUs), graphics processing units (GPUs), microprocessors, microcomputers, microcontrollers, digital signal processors, state machines, logic circuits, and/or any devices that manipulate signals based on operational instructions. As one example, the processor(s) 116 may include one or more hardware processors and/or logic circuits of any suitable type specifically programmed or configured to execute the algorithms and other processes described herein. The processor(s) 116 may be configured to fetch and execute computer-readable instructions stored in the computer-readable media 118, which may program the processor(s) 116 to perform the functions described herein. [0033] The computer-readable media 118 may include volatile and nonvolatile memory and/or removable and non-removable media implemented in any type of technology for storage of information, such as computer-readable instructions, data structures, programs, program modules, and other code or data. For example, the computer-readable media 118 may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, optical storage, solid state storage, magnetic disk, cloud storage, or any other medium that can be used to store the desired information and that can be accessed by a computing device. Depending on the configuration of the vehicle computing device(s) 104, the computer-readable media 118 may be a tangible non-transitory medium to the extent that, when mentioned, non-transitory computer- readable media exclude media such as energy, carrier signals, electromagnetic waves, and/or signals per se. In some cases, the computer-readable media 118 may be at the same location as the vehicle computing device 104, while in other examples, a portion of the computer-readable media 118 may be remote from the vehicle computing device 104.

[0034] The computer-readable media 118 may be used to store any number of functional components that are executable by the processor(s) 116. In many implementations, these functional components comprise instructions or programs that are executable by the processor(s) 116 and that, when executed, specifically program the processor(s) 116 to perform the actions attributed herein to the vehicle computing device 104. Functional components stored in the computer-readable media 118 may include the control program 124 and the image processing program 126, each of which may include one or more computer programs, applications, executable code, or portions thereof. Further, while these programs are illustrated together in this example, during use, some or all of these programs may be executed on separate computing devices such as separate vehicle computing device(s) 104. Alternatively, in some examples, the mobile device 112, such as a cellular smart phone, tablet computing device, wearable device, or the like, may execute the image processing program 126 and/or the control program 124, or portions thereof. As still another alternative, the smart eyewear 106 may execute the image processing program 126, or a portion thereof. As still another alternative, in some examples, the programs 124 and 126 may be part of a single program executed on a single computing device. Numerous other variations will be apparent to those of skill in the art having the benefit of the disclosure herein.

[0035] In addition, the computer-readable media 118 may store data, data structures, and other information used for performing the functions and services described herein. For example, the computer-readable media 118 may store operator data 128 that may be determined by the image processing program 126 and the control program 124 as discussed additionally below. In addition, the computer-readable media 118 may store one or more calibration images 129 as discussed below, e.g., with respect to FIG. 6, that may be used for determining whether a user’s FOV, head pose, etc., matches a recommended FOV or the like. The computing device(s) 104 may also include or maintain other functional components and data, which may include programs, drivers, etc., and the data used or generated by the functional components. Further, the computing device(s) 104 may include many other logical, programmatic, and physical components, of which those described above are merely examples that are related to the discussion herein.

[0036] The one or more communication interfaces 120 may include one or more software and hardware components for enabling communication with various other devices, such as through wired and/or wireless communications. For example, the communication interface(s) 120 may enable communication through one or more of a LAN, the Internet, cable networks, cellular networks, wireless networks (e.g., Wi-Fi) and wired networks (e.g., CAN, Fibre Channel, fiber optic, Ethernet), direct connections, as well as close-range radio communications such as

BLUETOOTH®, and the like, as additionally enumerated elsewhere herein.

[0037] The smart glasses 108 may include any of various different types of smart glasses, mixed reality glasses, and so forth. In the illustrated example, the smart glasses 108 include a forward-facing camera 130 and a projector or other display 132. In addition, the smart glasses 108 may include at least one rear-facing camera 134 that may be integrated into the same assembly as the display 132, or which may be separately positioned. The rear-facing camera 134 may be able to capture an image of at least one eye of a wearer of the smart glasses 108. In some examples, the smart glasses 108 include one or more processors, one or more computer-readable media, and one or more communication interface(s), each of which may be the same as or similar to those discussed above, respectively, with respect to the vehicle computing device(s) 104.

[0038] The smart glasses 108 are able to use the display 132 to add information to the real- world image that the wearer sees, such as to provide an augmented reality experience.

Furthermore, the forward facing camera 130 may be used to capture images of the real world that the wearer sees. Additionally, the rear facing camera 134 may be used to capture images of at least one eye of the wearer. In some examples, there may be two rear-facing cameras 134 positioned to capture images of both eyes of the operator 114. The images captured by the forward-facing camera 130 and rear-facing cameras 134 may be transferred to other devices, such as the mobile device 112 and/or the vehicle computing device 104, such as via Wi-Fi, BLUETOOTH®, cellular radio, or other communication technologies.

[0039] Furthermore, in addition to transferring data to other devices, the smart glasses 108 may be configured to receive information from other sources. For example, the smart glasses 108 may receive a communication from the vehicle computing device 104 or the mobile device 112, and may project or otherwise display an image based on the communication, such as to provide a notification to the operator 114. There are a number of different technologies currently in use for superimposing information onto the operator’s field of view (FOV) 135, such as an optical head- mounted display, embedded wireless glasses with transparent heads-up display, an augmented reality (AR) overlay, and so forth. Accordingly, implementations herein may employ any of these technologies to superimpose information in the wearer’s FOV 135. Implementations herein may use the images received from the operator’s smart glasses 108 (i.e., from both the forward facing camera 130 and the rear- facing camera(s) 134) and may present a notification or other information to the operator 114 via the smart glasses 108 to ensure safety of the vehicle operator 114 and the vehicle 102. Additionally, in the case of an emergency, the system 100 may send one or more control signals, such as to sound an alarm, provide haptic feedback via the smart glasses 108, alter an optical property for the smart glasses 108 (or for the smart contact lenses 110 discussed below), e.g., to show red symbol/area or provide other warning indicator.

[0040] In addition, the smart glasses 108 may include sensors, such as one or more accelerometers, gyroscopes, compasses, or the like, that may provide data for determining a wearer’s head posture. This sensor information may also be provided by the smart glasses 108 to the vehicle computing device 104, such as for use in determining the head posture of the user and for determining an appropriate action to perform for certain situations based on the sensor data as well as the image data and other information.

[0041] The smart contact lenses 110 may be similar to conventional contact lenses except that the smart contact lenses 110 may include a miniaturized camera 136 and other electronic circuitry

138, which may include storage, communication interface(s), and a microprocessor. Similar to a conventional contact lens, the smart contact lens 110 may be configured to sit on the cornea of the operator’s eye and may be removable when not in use. In some examples, the camera 136 may include auto focusing capability and further the smart contact lens 110 may include a projector or other display able to project an image directly onto a wearer’s retina or the like. The smart contact lenses 110 herein may be configured to capture images of what user sees in the real world, and further may capture some images the user’ s eyes, such as for detecting blinking rate, pupil dilation, partial eyelid closure conditions, and the like. The smart contact lenses 110 may wirelessly communicate these images to the mobile device 112 and/or the vehicle computing device 104, similar to the smart glasses 108 discussed above. In addition, the smart contact lenses 110 may receive data from other devices, such as for receiving a notification to be projected onto the retina of the operator 114, or otherwise displayed to the operator 114.

[0042] Furthermore, since the smart contact lenses 110 may not be able to capture an image of the operator’s eye(s), a driver- facing cabin-mounted camera 140 (such as a dashboard camera, visor camera, or the like) may be used to perform this function. In these examples, the images from the cabin-mounted camera 140 may be processed as discussed below concurrently with the images and/or other data from the smart contact lenses. The cabin-mounted camera 140 may also be used in combination with the smart glasses 108 in some cases, such as in the situation that the smart glasses 108 do not include a rear-facing camera.

[0043] FIG. 1 further illustrates examples of images captured of the FOV 135 of the operator 114 and further including overlays provided by the display capabilities of the smart eyewear 106. As a first example, an image 142 shows the FOV 135 of the operator 114 viewed looking forward through the windshield of the vehicle. An overlay 144 is presented by the smart eyewear overlying the FOV 135 of the operator 114 in real time to indicate to the operator the portion of the road to which the operator should be paying attention for navigating an upcoming right turn.

[0044] In a second example image 146, suppose that the system 100 has detected indications that the operator 114 is getting drowsy. For instance, as discussed additionally below, drowsiness may be detected based on a plurality of factors such as eyelid condition, operator head pose, operator FOV 135, and so forth. Based on detecting that the operator 114 is drowsy, the system 100 may cause a notification 148 to be presented on the FOV 135 of the operator 114, e.g., as illustrated. Further, in this situation, additional actions may be performed by the system, such as sounding an alert, causing a vibration of the eyewear 106 or other haptic output, causing other type of visual alerts, or the like.

[0045] Furthermore, should the system be unable to obtain images or other data from the smart eyewear 106, such as due to any technical issues (e.g., communication, sensor, etc.), the system may send an error message or other signal to the smart eyewear to alert the operator 114 of the failure. In addition, the alert may also be sent to one or more vehicle components, such as a vehicle dashboard display, vehicle speakers, or the like (not shown in FIG. 1), to take into account a situation in which all communication with the smart eyewear 106 has failed. [0046] In some examples, the vehicle 102 may also include onboard sensors 150 that may provide sensor information that may be used by the system, such as for localization of the vehicle for determining a recommended FOV for the operator 114 at any point during a trip. Examples of the onboard sensors 150 may include any of a plurality of different types of sensors such as a camera system, radar, lidar, ultrasound, a global navigation satellite system (GNSS) receiver (referred to hereinafter by the common usage name “GPS”, which is also intended to be inclusive of any other satellite navigation system), accelerometers, a compass, and the like. In addition, the sensor data may also include information received from or associated with various vehicle systems 122, such as from a suspension controller associated with the suspension system, a steering controller associated with the steering system, a vehicle speed controller associated with a braking and acceleration system, and so forth.

[0047] In some examples, the control program 124 may use rule-based and or artificial- intelligence-based control algorithms to determine parameters for vehicle control. For instance, the control program 124 may determine an appropriate action, such as braking, steering, accelerating, or the like, and may send one or more control signals to one or more vehicle systems 122 based on the determined action. For example, the control program 124 may send control signals to the suspension controller, the steering controller, and/or the vehicle speed controller for controlling or partially controlling the vehicle 102 in some applications.

[0048] FIGS. 2, 9A, 9B and 10-12 include flow diagrams illustrating example processes according to some implementations. The processes are illustrated as collections of blocks in logical flow diagrams, which represent a sequence of operations, some or all of which can be implemented in hardware, software or a combination thereof. In the context of software, the blocks may represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, program the processors to perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures and the like that perform particular functions or implement particular data types. The order in which the blocks are described should not be construed as a limitation. Any number of the described blocks can be combined in any order and/or in parallel to implement the process, or alternative processes, and not all of the blocks need be executed. For discussion purposes, the processes are described with reference to the environments, systems, and devices described in the examples herein, although the processes may be implemented in a wide variety of other environments, systems, and devices.

[0049] FIG. 2 is a flow diagram illustrating an example process 200 for monitoring an operator and determining an action according to some implementations. In some examples, the process 200 may be executed by the system 100 discussed above. For example, at least a portion of the process 200 may be executed by the vehicle computing device(s) 104, such as by execution of the image processing program 126 and the control program 124. In some cases, a portion of the process 200 may be executed by a mobile device 112 or the smart eyewear 106, as discussed above. Accordingly, the process 200 may be executed by a computing device that includes one or more of the vehicle computing device(s) 104, the mobile device 112, or the smart eyewear 106.

[0050] At 202, the computing device may receive an image of the operator’ s FOV and/or an image of the operator’s eye(s). For instance, the images may be received wirelessly from the smart eyewear such as via radio communication or other data communication technology. In some cases, the process 200 may be executed utilizing at least one image, e.g., either an image of the operator’s eye(s) or an image of the operator’s real-world FOV (i.e., what operator sees). In the example of FIG. 2, suppose that the computing device receives both an image of operator’s eyes and an image of what the operator sees.

[0051] At 204, the computing device may perform image preprocessing on the received images. For example, the image preprocessing may be performed to make the input image suitable for the processing using the consecutive image processing algorithms discussed below. Accordingly, several preprocessing operations may be performed dependent in part on the specifications of the cameras used to capture the images. Examples of the preprocessing operations may include changing of the image format, resizing the image, performing noise reduction on the image, performing channel separations, converting color to monochrome, performing histogram equalization, and so forth. The preprocessing operation may enhance significant image features and avoid unwanted distortions to enable the images to be used for the subsequent operations. In some cases, further preprocessing may be performed to create a stereo image, such as in the case in which there are two images corresponding to two FOV images from two different points of view (i.e., there may be multiple forward facing cameras to obtain two images representing what two eyes see).

[0052] At 206, the computing device may receive calibration image(s). For example, as discussed additionally below, e.g., with respect to FIG. 6, one or more calibration images for the operator 114 and the vehicle 102 may be stored in advance in the computer-readable media associated with the computing device 104.

[0053] At 208, the computing device may process the received FOV image and eye image using one or more image processing algorithms. For example, the image processing algorithms may configure the computing device to determine information such as subject identification, eye gage measurement, head pose measurement, distraction estimation, drowsiness, determine one or more vital signs, measure the operator’s fatigue level, and so forth, based on the pre-processed input images. Accordingly, the computing device may execute a plurality of algorithms using the FOV image and/or the eye image to perform the operations described in blocks 210-222 below. [0054] At 210, the computing device may identify the operator. For example, the operator’s eye image may be used to identify the operator. Based on the type and specifications of eyewear device, conventional iris recognition and/or eye vein recognition techniques may be employed. As one example, during the preprocessing steps, the computing device may determine that the eye image quality is good enough to perform iris recognition for identifying the operator 114. Additionally, or alternatively, if the image quality is sufficient, eye vein recognition may be used. As another alternative, operator identification using eye movement techniques may be applied to identify the operator.

[0055] At 212, the computing device may determine the operator’ s eye gaze. For example, the operator’s eye image may be used to measure or otherwise determine the operator’s eye gaze, e.g., a direction and focal point of the operator’ s gaze at the time when the image is captured. During this operation, the operator’ s eye is detected from the eye image using, such as by using traditional recognition techniques and/or using a trained machine-learning model, such a neural network, statistical model, or the like. After the eye has been recognized with respect to the entire image, key points and different features are identified in the image and the eye gaze is determined based on the extracted key point features. Additional details of determining eye gaze are discussed below with respect to FIGS. 3 A, 3B and 4, which include an example of key points and features that are extracted from the operator’s eye image.

[0056] At 214, the computing device may determine the operator’s head pose. For example, the operator’s head pose may be measured by analyzing the image of what the operator sees (operator real-world FOV image received from the forward facing camera). This image may be compared with a calibration image to estimate the operator’s head pose, such as the operator’s head roll, pitch, and yaw angles with respect to an XYZ coordinate system. Additional details of determining head pose are discussed below with respect to FIGS. 5 and 6.

[0057] At 216, the computing device may determine whether the operator is distracted. For example, operator distraction may include any activities that divert the operator’ s attention from driving. Similar to the head pose estimation technique discussed above, the operator’s distraction may be determined by comparing the received FOV image that indicates what the operator sees with a calibration image. Additional details of determining operator distraction are discussed below with respect to FIGS. 5, 6, 7 A and 7B. [0058] At 218, the computing device may determine whether the operator is drowsy. For example, drowsiness may correspond to a state of sleepiness and may be directly related to the degree of alertness of the operator. The drowsiness in some examples herein may be determined based on eye-related features. As one example, drowsiness may be determined based on percentage of eye closure (PERCLOS). For example, studies have shown that operators with acute sleep deprivation showed increasing frequency and duration of episodes of prolonged eyelid closure. Accordingly, some examples herein perform drowsiness estimation of the operator based on eye closure detection over time for a sequence of images of the operator’s eye, e.g., captured at a known frequency, such as several images per second, or the like.

[0059] As discussed additionally below with respect to the eye feature estimation of FIG. 3 A, key point features of the operator’s eye are extracted from each eye image in a sequence of consecutive eye images. The average value of the distances between pairs of eye key points (e.g., two pairs for each eye) may be employed as an evaluation criteria. As one example, if the mean value is above 10 pixels, then both eyes may be considered as open, otherwise closed. In addition, the PERCLOS may be stored over several consecutive image frames for further processing to determine a length of time the eyes are closed if they are determined to be closed.

[0060] As one example, based on the Boolean eye closure values (opened or closed) for each image frame, the eye blinking distance (EBD) may be calculated. The EBD may include the time interval in seconds between two eye closures. An eye-opening (EO) feature may correspond to PERCLOS over the most recent plurality of frames (e.g., 30 frames or other suitable number depending on sampling rate), where PERCLOS is calculated based on:

where, x is the distance between upper and lower eyelid, subscript i indicates frame number, and left or right indicates either left eye or right eye. The following rules may be applied for ternary drowsiness classification:

IF the value of EO is below 80%, THEN the state of drowsiness is considered ‘drowsy’;

IF EBD is larger than 15 seconds, THEN the state of drowsiness is considered ‘starring’ ;

IF none of the above-mentioned conditions is true, then the drowsiness state is considered ‘operator not drowsy’. [0061] At 220, the computing device may determine vital signs of the operator. For example, medical signs that indicate the state of a patient’s vital body functions are knows as vital signs, such as temperature, blood pressure, respiration rate, and heart rate. In some examples herein, the operator’s respiratory rate and heart rate may be estimated based at least on the eye image captured by the smart eyewear. As one example, for heart rate measurement, the operator’s blood volume pulse (BVP) signal may be detected using image analysis of face skin (e.g., skin nearby the operator’s eye) over a sequence of consecutive images. For instance, a consistent region of interest may be selected from each eye image in the sequence of images. Features from the images may be extracted, such as using both an intensity-based and motion-based approach. After eliminating the noise and dimensionality deductions, a cardiac signal may be obtained. After the cardiac signal is obtained, at least one of the two approaches - namely, time domain approach and frequency domain approach may be used to calculate the heart rate based on the sequence of images. In the time domain approach, a peak detection may be applied to calculate the inter-beat interval that ultimately indicates the heart rate. On the other hand, in the frequency domain approach, the power spectral density may be analyzed, where the dominant frequency indicates the heart rate.

Additional details of determining heart rate using eye images are discussed below with respect to

FIG. 9A. Respiration frequency may be calculated using a similar technique.

[0062] At 222, the computing device may determine whether the operator is fatigued. As one example, operator fatigue may be estimated using the eye image captured by smart wearable device. As one example, the operator’ s eye pupil diameter and eye blinking rate may be used to estimate the fatigue or stress level of the operator. For instance, the level of attentiveness and stress level of an operator are may be expressed by eye blinking rate and pupil diameter. Fatigue and drowsiness results significant changes in eye blinking pattern as well as pupil diameter. In some examples, a consecutive sequence of the operator’s eye images may be acquired from the smart eyewear device and may be preprocessed as explained above. After the operator’s eye is detected from an entire image, the key point features and pupil diameter may be identified/measured as shown in FIG. 3 A. The average value of the distances between pairs of key points (e.g., two pairs for each eye) may be used as an evaluation criteria. Similar to the evaluation discussed above with respect to drowsiness, if this mean value is above, e.g., 10 pixels, then both eyes may be considered as open, otherwise closed. Furthermore, the eye blinking rate may be measured in blinks/min and may correspond to the number of blinks occurring during a certain period. The eye blinking rate and pupil diameters may be used as input indicators for a fatigue estimation module. This fatigue estimation module may be configured using a combination of data driven and knowledgebase approaches. For instance, studies have been conducted to measure eye blink rate and pupil diameter for many subjects considering different age, race, location, condition, etc., and for different stress levels. These empirical results have been stored to create a database. This database may be used to design fatigue estimation algorithm using both machine learning and rule-based methodologies. FIG. 9B discussed below include a flowchart providing additional details of the fatigue estimation techniques herein. As one example, the operator fatigue level may be categorized as excellent, very good, good, average, fair, poor, and very poor.

[0063] At 224, the computing device may rate the operator’ s physical condition and level of attention. For example, operator’s categorized eye gaze, head pose, distraction, drowsiness, vital signs, and fatigue levels may be used to design a rule based controller to estimate the operator’s physical condition and attention level. A fuzzy rule-based algorithm may be employed to measure operator’s physical condition and attention level. Furthermore, a set of IF-THEN-ELSE rules may be applied to output the operator’s physical condition and attention level. Operator’s physical condition and attention level may be categorized into excellent, very good, good, average, fair, poor, and very poor conditions.

[0064] During one sampling time ti, after inputs are received, the image processing program may apply the evaluation rules and may combine results from each rule. During defuzzification the image processing program may be executed to convert the determined values into a physical condition level and attention level for the operator. The determined physical condition and attention level for the operator may be sent to the eyewear or other display for presentation to the operator are transferred to display to the operator as well as stored by the computing device and/or sent to a service computing device over a network for storage at a network location. Additionally, in other examples, machine-learning algorithms including artificial neural networks or other types of algorithms may be used to determine the operator’s physiological condition and attention level.

[0065] At 226, the computing device may compare the operator’s current FOV with a recommended FOV for the current location of the vehicle. For example, the operator’s head posture and eye gaze may be used to estimate the operator’s FOV. As one example, the operator’s head roll, head pitch, and head yaw angles may be determined, as well as the operator’s horizontal and vertical eye gaze angles. For instance, the system may determine the vehicle location (e.g., based on received GPS information and/or landmark information received from the vehicle sensors) and may perform localization of the vehicle to determine the vehicle on a location map, such as a high definition (HD) map. In addition, a recommended FOV may be estimated using the current vehicle location, an intended destination, surround landmarks, surrounding obstacles, etc.

The recommended FOV e.g., expressed in terms of an XYZ global coordinate system, local coordinate system, or the like, may be is estimated. Further, the operator’s current FOV may be is estimated using 3D head pose and eye gaze angles, and may be plotted on the current location map. Additionally, the computing device may compare the estimated FOV of the operator with the recommended FOV.

[0066] At 228, the computing device may determine whether the operator’s estimated FOV is within a threshold amount of the recommended FOV. If not, the process goes to 230. If so, the process returns to 202 to process a next image.

[0067] At 230, the computing device may determine whether the operator’s estimated FOV is outside the threshold of the recommended FOV for more than a threshold period of time. For example, if the operator’s estimated FOV is not within the threshold amount of the recommended FOV for at least a threshold period of time, the computing device may determine that the operator is not looking in the recommended direction and may be jeopardizing the safety of the operator and/or the vehicle. For example, if the operator quickly glances away from the recommended FOV, an alert may not be generated based on a single image. However, if consecutive images indicate that the operator is not looking in the recommended direction for a threshold time, a warning signal may be sent to the smart eyewear and/or to vehicle devices, such as a vehicle display, speaker, or the like. Accordingly, if the operator’s FOV is not outside the threshold for the threshold period of time the process may return to 202. On the other hand, if the operator’s FOV is outside the threshold for the threshold period of time then the process goes to 232 to generate an alert or other notification.

[0068] At 232, the computing device may send a message or other notification to present to the operator or take other action. For example, the proposed system can estimate whether the operator is able to operate the vehicle safely or not utilizing operator’s current FOV and physical condition and attention level data. As several examples, the computing device may send a signal to vibrate or provide other haptic feedback to the operator, may cause the smart eyewear to generate a sound, and/or may cause the vehicle devices to generate sounds or visual alerts. Additionally, in the case of a smart contact lens, the computing device may send a signal to cause a change in the color of the contact lens (e.g., lenses becomes red) or the like, to warn the operator. Additionally, or alternatively, the computing device may send the recommended FOV coordinate value to the wearable device to display the recommended FOV on the device, e.g., as an overlay as shown above in FIG. 1 at image 142.

[0069] In addition, if the computing device determines that the detects the operator is not able to operate the vehicle safely, the computing device may send one or more control signals to the vehicle systems 122 such as to take over control of the vehicle for driving and/or parking the vehicle autonomously. In some examples, the computing device may communicate over one or more networks connects to with a service computing device to store operator’s driving and physical condition data which may be shared and/or utilized by other service providers.

[0070] FIGS. 3A and 3B illustrate examples of determining eye gaze according to some implementations. FIG. 3A illustrates an example 300 of feature estimation according to some implementations. FIG. 3A may correspond, e.g., to portions of blocks 202, 204, 208-212, and 218- 222 of FIG. 2.

[0071] At 302, an image of the operative is received e.g., as discussed above with respect to block 202 of FIG. 2.

[0072] At 304, preprocessing may be performed on the received image as discussed above, e.g., with respect to block 204 of FIG. 2.

[0073] At 306, the eye is detected from the entire image, the key point features may be extracted, and the eye gaze may then be determined, e.g., as discussed with respect to FIG. 3B. For instance, a plurality of key point features 308 may be identified around the perimeter of the eye. Further, the location of the iris 310 may be identified. In addition, the location of the pupil may be determined and a diameter D1 of the pupil may be determined. In some cases, the operations illustrated in FIG. 3A might not be completely separated from subject identification procedure of block 210 discussed above with respect to FIG. 2. For instance, operator identification techniques such as eye movement/iris recognition/eye vein recognition may be employed after the eye features of the iris, the white region, and the pupil are identified in the feature estimation operation of 306.

[0074] FIG. 3B illustrates examples 320 of eye gaze estimation according to some implementations. For example, eye gaze estimation may include determination of an angle and/or focal point of the operator’s eyes. Implementations herein are able to estimate eye gaze angle and focal point accurately in both horizontal and vertical planes. For instance, as illustrated at examples 322, 324 and 326 the operator’s eye gaze angles can be estimated for both horizontal and vertical planes, such as in degrees. For example, as indicated at 322 a focal point 330 of the operator 114 may be determined relative to a head position of the operator 114. Furthermore, as indicated at 324 a horizontal gaze angle may be determined for the operator’s eyes based on relative angles A1 and A2. Furthermore, as indicated at 326 a vertical gaze angle A3 may be determined for the operator’s eyes. Once the eye gaze angles are estimated, the implementations herein may also categorize the eye gaze in terms of linguistic variables for both of the horizontal and vertical planes, such as the following:

[0075] Eye Gaze Horizontal = {Large negative (LN), Medium Negative (MN), Small (S), Medium Positive (MP), Large Positive (LP)} [0076] Eye Gaze Vertical = {Large negative (LN), Medium Negative (MN), Small (S), Medium Positive (MP), Large Positive (LP).

[0077] PIG. 4 illustrates an example of a membership function 400 for determining horizontal eye gaze according to some implementations. It can be noted that the number of linguistic variables as well as range (of values) of linguistic variables can be adjusted based on the different conditions and subjects. The system also generates membership function of the variables for eye gaze for both planes. Commonly available functions including triangular, trapezoidal, gaussian, sigmoidal, etc., functions that may be used to compose the membership function.

[0078] In the illustrated example, such as based on example 322 of PIG. 3B, the membership function 400 includes the following categories {Large negative (LN), Medium Negative (MN), Small (S), Medium Positive (MP), Large Positive (LP)} for horizontal eye gaze, where LN corresponds to minus 50 degrees, MN corresponds to -25 degrees, S corresponds to 0 degrees, MP corresponds to 25 degrees, and a LP corresponds to 50 degrees. Purthermore, in other examples, different ranges of degree values may be used for the various categories. In addition, the membership function for the vertical eye gaze may be similarly determined.

[0079] FIG. 5 illustrates an example of a head pose coordinate system 500 according to some implementations. Lor example, the operator’s head pose may be measured by analyzing the POV image that indicates what the operator sees, and this analysis may be augmented by receiving accelerometer, gyroscopic, and/or compass sensor data from the smart eyewear. In some examples, the operator POV image may be compared with a calibration image to estimate operator’s head pose. Lor instance, a calibration image may be captured for the operator with a known head pose before the operating the vehicle using the smart eyewear device. One or more calibration images 129 for the operator and the vehicle may be stored in the computer-readable media, e.g., as discussed above with respect to PIG. 1. An example calibration image 129 is shown in FIG. 6. The features from the calibration image 129 may be extracted using a feature extraction algorithm a as discussed additionally below. The image with marked image features may be saved as the calibration image 129.

[0080] In the example of FIG. 5, the head angles of the operator may be estimated in 3D space, namely head yaw, head pitch, and head roll angles in an XYZ coordinate system, as illustrated. Accordingly, the operator’s head pose may be measured by comparing the operator FOV image with the calibration image to estimate operator’s head pose in each of the angular degrees of freedom. After the 3D head angles are estimated, the proposed system may categorize the angles in linguistic variables similar to the technique discussed above with respect to FIG. 4, e.g.: [0081] Head Yaw Angle = {Large negative (LN), Medium Negative (MN), Small (S), Medium Positive (MP), Large Positive (LP)};

[0082] Head Roll Angle = {Large negative (LN), Medium Negative (MN), Small (S), Medium Positive (MP), Large Positive (LP)};

[0083] Head Pitch Angle = {Large negative (LN), Medium Negative (MN), Small (S), Medium Positive (MP), Large Positive (LP).

[0084] In the examples herein, the number of linguistic variables as well as the range (of values) of linguistic variables may be adjusted based on the different conditions and subjects. The system may also generate membership functions of the variables for three different head angles. [0085] FIG. 6 illustrates an example of the calibration image 129 according to some implementations. For example, the calibration image may be based on an operator FOV image captured when the operator is looking straight ahead out of the windshield of the vehicle, or the like. The calibration image may then be processed to extract features such as based on edge recognition, or the like, and the extracted features may be identified in the image as indicated at 602. The calibration image may then be stored and used for determining the variations in the operator’s head pose in subsequently captured operator FOV images. In some examples, a conventional image processing feature matching algorithm and/or a deep convolutional neural network-based algorithm may be used to match the features of a current FOV image with the calibration FOV image. Further, the number of image features 602 may be varied based on image sensor specifications and/or computational power specifications of the computing device that is being used to perform the comparison.

[0086] FIGS. 7A and 7B illustrate examples of detecting distraction of the operator according to some implementations. For example, operator distraction may include activities that divert operator’s attention from operating the vehicle. Similar to the head pose estimation technique discussed above, the vehicle operator’ s distraction may be determined at least in part by comparing a current FOV image of what the operator sees with the calibration image 129, e.g., as discussed above with respect to FIG. 6. For example, it is evident that all of the features of calibration image may not be available in the images of distraction driving, or additional elements may be present.

[0087] FIG. 7A illustrates an example 700 of detecting distraction by detecting that the operator is looking out the side window according to some implementations. For instance, system may receive an image 702 from the forward facing camera of the smart eyewear indicating the operator’s current FOV. The operator’s level of distraction may be estimated based on the number of features missing and the current features included in comparison with the calibration image feature locations. Thus, in the example of FIG. 7A, the side view mirror and other features may be included in the FOV image 702 while numerous other features from the calibration image are not included.

[0088] Furthermore, the operator distraction may not be decided based on a single FOV image. For example, during a turn or lane change, the operator should look to the right or left side for safety purposes. However, if a plurality of consecutive images (e.g., for a time > threshold time) indicates that the operator FOV does not match with the calibration image features, the system may determine that the operator is distracted. In some examples, the system may categorize the level distraction as discussed below with respect to FIG. 8, and may use a category for determining whether to present an alert or other notification to the operator.

[0089] FIG. 7B illustrates an example 710 of detecting distraction according to some implementations. In this example, an additional element is included in the current FOV image 712. For example, the system may be configured to detect certain items that may cause distraction such as mobile devices or other items may be held in the hand of an operator. Accordingly, the system may increase the categorization of the level distraction based on the recognition of the additional element 714 when comparing the current FOV image 712 with the calibration image. [0090] FIG. 8 illustrates an example membership function 800 for distraction variables according to some implementations. For example, the system may categorize the level of distraction into following linguistic variables and may generate the illustrated membership function 800: Distraction = {Extreme (E), Very High (VH), High (H), Medium (M), Low (D), OK (O)}. For instance, Xi through X₅ may indicate the number of features in the current FOV image that match with the calibration image. Furthermore, the addition of a recognized distraction, such as the mobile devices discussed above with respect to FIG. 7B, may act as a negative count against the number matching features.

[0091] FIGS. 9A and 9B are flow diagrams illustrating example processes 900 and 920, respectively, for determining physiological conditions of an operator according to some implementations. In some examples, the processes 900 and 920 may correspond at least in part to blocks 220 and 222, respectively, of the process 200 of FIG. 2 discussed above.

[0092] FIG. 9A is a flow diagram illustrating the example process 900 for determining a heart rate of an operator according to some implementations. As mentioned above, temperature, blood pressure, respiration rate, and heart rate are included in vital signs that may be monitored for a vehicle operator. In some examples herein, the operator’ s respiratory rate and heart rate may be determined using an eye image, such as may be captured by a rear facing camera of the smart glasses or the like. For heart rate measurement, operator’s blood volume pulse (BVP) signal may be determined using image analysis of face skin (skin nearby of eye) for a sequence of consecutive images of the operator’s eye.

[0093] At 902, the system may receive an image of the operator’s eye. This operation may correspond to block 202 discussed above with respect to FIG. 2.

[0094] At 904, the system may perform image preprocessing on the received image. This operation may correspond to block 204 discussed above with respect to FIG. 2.

[0095] At 906, the system may determine a region of interest (e.g., a patch of skin near the eye) for performing the analysis. The same region of interest may be used for the plurality of consecutive images.

[0096] At 908, system may perform analysis on the region of interest over the plurality of consecutive images to recognize a heartbeat of the operator. At least one of intensity-based analysis 910 (e.g., changes in image intensity at the region of interest) or motion based analysis 912 (e.g., movement of the skin at the region of interest) may be applied.

[0097] At 914, the system may extract a cardiac signal based on the changes detected over the series of consecutive images. For example, after eliminating the noise and dimensionality deductions, the cardiac signal may be obtained from detected changes in the images.

[0098] At 916, the system may estimate the heart rate of the operator. For example, after the cardiac signal is extracted, at least one of the two approaches, namely, time domain approach or frequency domain approach may be used to calculate the heart rate. In the time domain approach, peak detection may be applied to calculate an inter-beat interval which may be used to ultimately indicate the heart rate. On the other hand, in frequency domain approach, power spectral density may be analyzed to detect changes, where the dominant frequency indicates the heart rate. As one example, the computing device may be configured to categorized heart rate into following linguistic variables and generate a membership function accordingly: Heart Rate = {Excellent (E),

Very Good (VG), Good (G), Average (A), Fair (F), Poor (P), Very Poor (VP).

[0099] FIG. 9B is a flow diagram illustrating the example process 920 for determining a fatigue level of an operator according to some implementations. As mentioned above, the operator fatigue may be estimated using the eye image captured by the rear-facing camera of smart eyewear, or the like. For example, the operator’s eye pupil diameter and eye blinking rate may be used to estimate the fatigue or stress level. Fatigue and drowsiness may result in significant changes in eye blinking pattern as well as pupil diameter. In some implementations herein, the operator’s eye image may be acquired and preprocessed as previously explained above.

[00100] At 922, the system may receive an image of the operator’s eye. This operation may correspond to block 202 discussed above with respect to FIG. 2. [00101] At 924, the system may perform image preprocessing on the received image. This operation may correspond to block 204 discussed above with respect to FIG. 2.

[00102] At 926, the system may detect the pupil and key point features in the eye image. An example of this operation is discussed above, e.g., with respect to FIG. 3 A. For instance, after the eye is detected from the entire preprocessed image, key point features and pupil diameter may be measured.

[00103] At 928, the system may perform analysis based on the detected pupil and key point features. For instance, the analysis may include determining an eye blinking rate, as indicated at 930, and determining a pupil diameter, as indicated at 932. The average value of the distances between pairs of eye key point features (e.g., two pairs for each eye) may be used as an evaluation criteria. As one example, if the mean value is above 10 pixels, then both eyes may be considered as open, or if not, then closed. The eye blinking rate may also be determined, e.g., blinks/min and corresponding to the number of blinks occurring during a certain period. The eye blinking rate and pupil diameter may be used as input information for performing the fatigue estimation. [00104] At 934, the system may input the eye blinking rate and pupil diameter into a fatigue estimation algorithm. As mentioned above, in some examples, a database of empirically determined eye blink rates and pupil diameters for many subjects considering different age, race, location, condition, etc. for different stress level may be accessed. The database may be used when configuring the fatigue estimation algorithm, such as based on at least one of artificial intelligence (AI) and mle -based methodologies.

[00105] At 936, based on the fatigue estimation algorithm, the fatigue level of the operator may be categorized as one of excellent, very good, good, average, fair, poor, and very poor.

[00106] FIG. 10 illustrates an example architecture of a system 1000 and operations performed by the system according to some implementations. For example, the system 1000 employed in FIG. 10 may correspond in some examples to the system 100 discussed above with respect to FIG. 1. In this example, the vehicle computing device 104 (or, in some examples, the mobile computing device 112) may receive information from the smart eyewear 106. For example, the vehicle computing device 104 may receive FOV images 1002 of what the operator 114 sees and/or may receive eye images 1004 of the operator’s eyes.

[00107] At 1006, the image processing program 126 may be executed to perform image pre processing. For example, the image preprocessing may be performed as discussed above, space e.g., at block 204 of FIG. 2.

[00108] At 1008, the image processing program 126 may be executed to analyze the received images. For instance, the analysis may be performed to identify the operator as indicated at 1010, determine the eye gaze of the operator as indicated at 1012, determine the head pose of the operator as indicated at 1014, determine distraction of the operator as indicated at 1016, determine drowsiness of the operator as indicated at 1018, determine vital signs of the operator as indicated at 1020, and/or determine fatigue levels of the operator as indicated at 1022. Description of these operations are discussed above, e.g., with respect to blocks 208-222 of FIG. 2.

[00109] At 1024, the image processing program 126 may be executed to provide the analysis results to the control program 124. For example, the control program 124 may be executed on the vehicle computing device 104 or other suitable computing device as discussed elsewhere herein. [00110] At 1026, the control program may be executed to determine one or more actions based on the analysis results received from the image processing program 126. Blocks 1028, 1030, and 1032 provide examples of actions that may be performed based on the analysis results.

[00111] At 1028, the computing device may execute the control program 124 to determine the operator’s physical condition and level of attention. For instance, as discussed above, e.g., with respect to FIG. 2, the operator’s categorized eye gaze, head pose, distraction, drowsiness, vital signs, and fatigue levels may be used to estimate operator’s physical condition and attention level based on a set of rules. As one example, a fuzzy-rule-based framework may be used to determine the operator’s physical condition and attention level. A set of IF-THEN-ELSE rules may be applied to output the operator’s physical condition and attention level. As one example, the operator’s physical condition and attention level may be categorized into one of the following categories: excellent, very good, good, average, fair, poor, and very poor. During one sampling time ti, after the inputs are received, the control program 124 may evaluate the analysis results based on the rules and may combine the results from each rule. The control program 124 may further perform defuzzification, such as by converting the results into a physical condition level and an attention level. Alternatively, other AI algorithms including artificial neural networks may be used to measure operator’ s physical condition and attention level.

[00112] In some examples, as indicated at 1034, the control program 124 may send the operator’s determined physical condition and attention level as a notification for presentation on a display 1036 of the vehicle. Additionally, or alternatively, as indicated at 1038, the control program 124 may send the notification for presentation on the eyewear display. Furthermore, as indicated at 1040, the control program 124 may send the information determined about the operator to a service computing device 1042 over one or more networks 1044.

[00113] At 1030, the computing device may compare the operator’s FOV, as captured by the eyewear, with the recommended FOV. The vehicle operator’s head pose and eye gaze may be used along with a current operator FOV image 1002 received from the smart eyewear 106 to estimate the operator’ s FOV. As mentioned above, the head roll, head pitch, and head yaw angles as well as horizontal and vertical eye gaze angles determined from the eye images 1004 may be used in this stage. Furthermore, the control program 124 may be executed to identify the current vehicle location, such as based on received GPS information 1046, and may perform localization to determine the current location of the vehicle with respect to a location map, such as an HD map or the like.

[00114] Furthermore, based on the determined current location of the vehicle, the control program 124 may obtain location FOV information 1048 from a location FOV database 1050. For example, the FOV database 1050 may include recommended FOV for the roads on which the vehicle is expected to travel and may be accessed by a large number of vehicles during navigation. In some examples, the location FOV database 1050 may be accessed over the one or more networks 1044, while in other examples, at least a portion of the FOV database 1050 may be maintained in a local storage accessible by the vehicle computing device 104. The recommended FOV may be estimated using current location, destination, surrounding obstacles, etc. The recommended FOV may be estimated considering both local and global co-ordinate systems. [00115] At 1032, based on the comparison of the FOV of the operator with the recommended FOV, if the difference exceeds a threshold value, a warning signal or other notification may be sent to the vehicle display 1036 and/or to the headwear 106. Additionally, or alternatively, the control program may send a signal to cause the headwear to vibrate, make a sound, or provide various alerts or other notifications as discussed above. In addition, the control program may send information related to the recommended FOV coordinate value to the smart eyewear 106, such as to cause the smart eyewear 106 to display the recommended FOV on the device, such as in the form of an overlay or the like as discussed above e.g., with respect to FIG. 1. Furthermore, if the control program 124 determines that the operator is not physically able to safely operate the vehicle or is in danger of an accident, the control program 124 may send a control signal 1052 to one or more of the vehicle systems 122, such as to cause the vehicle systems to perform auto park, or other AD/ AD AS functions, as indicated at 1054.

[00116] The service computing device(s) 1042 may include one or more servers or other types of computing devices that may be embodied in any number of ways. For instance, in the case of a server, the programs, other functional components, and data may be implemented on a single server, multiple servers, a cluster of servers, a server farm or data center, a cloud-hosted computing service, and so forth, although other computer architectures may additionally or alternatively be used. Multiple service computing device(s) 1042 may be located together or separately, and organized, for example, as virtual servers, server banks, and/or server farms. The described functionality may be provided by the servers of a single entity or enterprise, or may be provided by the servers and/or services of multiple different entities or enterprises.

[00117] In the illustrated example, the service computing device(s) 1042 includes, or may have associated therewith, one or more processors 1060, one or more computer-readable media 1062, and one or more communication interfaces 1064. Each processor 1060 may be a single processing unit or a number of processing units, and may include single or multiple computing units, or multiple processing cores. The processor(s) 1060 can be implemented as one or more central processing units, microprocessors, microcomputers, microcontrollers, digital signal processors, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. As one example, the processor(s) 1060 may include one or more hardware processors and/or logic circuits of any suitable type specifically programmed or configured to execute the algorithms and processes described herein. The processor(s) 1060 may be configured to fetch and execute computer-readable instructions stored in the computer-readable media 1062, which may program the processor(s) 1060 to perform the functions described herein.

[00118] The computer-readable media 1062 may include volatile and nonvolatile memory and/or removable and non-removable media implemented in any type of technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. For example, the computer-readable media 1062 may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, optical storage, solid state storage, magnetic tape, magnetic disk storage, storage arrays, network attached storage, storage area networks, cloud storage, or any other medium that can be used to store the desired information and that can be accessed by a computing device. Depending on the configuration of the service computing device(s) 1042, the computer-readable media 1062 may be a tangible non-transitory medium to the extent that, when mentioned, non-transitory computer-readable media exclude media such as energy, carrier signals, electromagnetic waves, and/or signals per se. In some cases, the computer-readable media 1062 may be at the same location as the service computing device 1042, while in other examples, the computer-readable media 1062 may be partially remote from the service computing device 1042.

[00119] The computer-readable media 1062 may be used to store any number of functional components that are executable by the processor(s) 1060. In many implementations, these functional components comprise instructions or programs that are executable by the processor(s) 1060 and that, when executed, specifically program the processor(s) 1060 to perform the actions attributed herein to the service computing device(s) 1042. Functional components stored in the computer-readable media 1062 may include a storage program 1066 which may include one or more computer programs, applications, executable code, or portions thereof.

[00120] In addition, the computer-readable media 1062 may store data, data structures, and other information used for performing the functions and services described herein. For example, the computer-readable media 1062 may store the operator rating data 1068, and operator health information 1070. The service computing device 1042 may also include or maintain other programs and data, which may include various programs, operators, etc., and the data used or generated by the functional components. Further, the service computing device 1042 may include many other logical, programmatic, and physical components, of which those described above are merely examples that are related to the discussion herein.

[00121] The one or more communication interfaces 1064 may include one or more software and hardware components for enabling communication with various other devices, such as over the one or more network(s) 1044. For example, the communication interface(s) 1064 may enable communication through one or more of a LAN, the Internet, cable networks, cellular networks, wireless networks (e.g., Wi-Fi) and wired networks (e.g., Fibre Channel, fiber optic, Ethernet), direct connections, as well as close-range communications such as BLUETOOTH®, and the like, as additionally enumerated elsewhere herein.

[00122] The one or more networks 1044 may include any appropriate network, including a wireless network, such as a cellular network; a wide area network, such as the Internet; a local area network, such an intranet; a local wireless network, such as Wi-Fi; close-range wireless communications, such as BLUETOOTH®; a wired network, including fiber optics and Ethernet; any combination thereof, or any other suitable communication network. Components used for such communication technologies can depend at least in part upon the type of network, the environment selected, or both. Protocols for communicating over such networks are well known and will not be discussed herein in detail.

[00123] FIG. 11 is a flow diagram illustrating an example process 1100 for monitoring an operator according to some implementations. In some examples, the process 1100 may be executed by the system 100 and/or 1000 discussed above. For example, the process 1100 may be executed by the computing device(s) discussed above such as by executing the control program 124 and image processing program 126 in some examples. In addition, in some cases, the process 1100 may correspond in part to blocks 224-232 of the process 200 of FIG. 2 discussed above. [00124] At 1102, the computing device may determine a destination for the vehicle. For example, the computing device determine whether a destination has been provided to a navigation portion of the control program or the like. [00125] At 1104, the computing device may determine whether the destination has been specified. If so, the process goes to blocks 1108 and 1114. If not, the process goes to 1106. [00126] At 1106, the computing device may predict a destination if the destination is unknown. For example, based on a current heading of the vehicle and based on access to a list of past destinations, the vehicle computing device may predict the current destination. Additionally, or alternatively, the vehicle computing device may query the operator for the destination.

[00127] At 1108, the computing device may determine a current location of the vehicle. For example, the computing device may receive GPS information from a GPS receiver.

[00128] At 1110, the computing device may use the GPS information, landmark information, etc., to localize the vehicle with respect to a map such as an HD map or the like.

[00129] At 1112, the computing device may determine a recommended FOV for the current vehicle location. For instance, as discussed above, the computing device may obtain the recommended FOV from an FOV database or the like.

[00130] At 1114, the computing device may determine an operator head pose and eye gaze. Techniques for making these determinations are discussed above e.g., with respect to FIGS. 2-6. [00131] At 1116, the computing device may determine the operator’s current FOV such as based on an FOV image received from the smart headwear.

[00132] At 1118, the computing device may compare the operator FOV with the recommended FOV. This operation is discussed above e.g. with respect to FIGS. 2, 6-8 and 10.

[00133] At 1120, the computing device may determine whether the difference between the operator FOV and the recommended FOV exceeds a threshold period if not, the process returns to 1114 to process a next image. If so, the process goes to 1122.

[00134] At 1122, the computing device may determine whether the threshold has been exceeded for a threshold amount of time. If not, the process goes to 1114 to process a next image. If so, the process goes to 1124.

[00135] At 1124, when the threshold difference between the operator FOV and the recommended FOV has exceeded the threshold for a threshold amount of time, the computing device may send a signal to display the recommended FOV on the eyewear and/or other display such as the vehicle display. In addition, or alternatively, the computing device may send a notification or provide other feedback to the operator such as to direct the operator’ s attention to the recommended field of view. Examples of types of feedback are discussed above e.g. with respect to FIGS. 1-10.

[00136] FIG. 12 is a flow diagram illustrating an example process 1200 for monitoring an operator and/or controlling a vehicle according to some implementations. In some examples, the process 1200 may be executed by the system 100 and/or 1000 discussed above. For example, the process 1200 may be executed by the computing device(s) discussed above such as by executing the control program 124 and image processing program 126 in some examples. In addition, in some cases, the process 1200 may correspond in part to blocks 202-232 of the process 200 of FIG. 2 discussed above.

[00137] At 1202, the computing device may determine a destination for the vehicle. For example, the computing device determine whether a destination has been provided to a navigation portion of the control program or the like.

[00138] At 1204, the computing device may determine whether the destination has been specified. If so, the process goes to block 1208. If not, the process goes to 1206.

[00139] At 1206, the computing device may predict a destination if the destination is unknown. For example, based on a current heading of the vehicle and based on access to a list of past destinations, or the like, the vehicle computing device may predict the current destination. Additionally, or alternatively, the vehicle computing device may query the operator for the destination.

[00140] At 1208, the computing device may receive an image of the operator FOV and in some cases an image of the operator’s eye(s).

[00141] At 1210, the computing device may perform preprocessing on the received images. [00142] At 1212, the computing device may analyze the received images. Examples of analyzing the images are discussed above such as at blocks 208-222 of FIG. 2.

[00143] At 1214, the computing device may determine parameters based on the image analysis. For instance, as discussed above, e.g., with respect to blocks 208-222 of FIG. 2, the computing device may determine an identity of the operator, operator eye gaze, operator head pose, operator distraction, operator drowsiness, operator vital signs, and/or operator fatigue.

[00144] At 1216, the computing device may determine linguistic variables and membership functions for the determined parameters. For example, as discussed above e.g. with respect to FIGS. 2-10, the computing device may classify the determined parameters according to specified categories based on application of membership functions or the like.

[00145] At 1218, the computing device may compare the operator FOV with the recommended FOV. This operation is discussed above e.g. with respect to FIGS. 2, 6-8 and 10.

[00146] At 1220, the computing device may determine whether the difference between the operator FOV and the recommended FOV exceeds a threshold period if not, the process returns to 1208 to process a next image. If so, the process goes to 1122. [00147] At 1222, the computing device may determine whether the threshold has been exceeded for a threshold amount of time. If not, the process goes to 1208 to process a next image. If so, the process goes to 1124.

[00148] At 1124, when the threshold difference between the operator FOV and the recommended FOV has exceeded the threshold for a threshold amount of time, the computing device may send a signal to display the recommended FOV on the eyewear and/or other display such as the vehicle display. In addition, or alternatively, the computing device may send a notification or provide other feedback to the operator such as to direct the operator’ s attention to the recommended field of view. Examples of types of feedback are discussed above e.g. with respect to FIGS. 1-10.

[00149] At 1228, the computing device may determine the operator’s physical condition and attention level based on the analysis of the received images.

[00150] At 1230, the computing device may determine whether the operator is able to operate the vehicle safely. As mentioned above, the system may estimate whether the operator is able to operate the vehicle safely or not utilizing operator’s field of view, physical condition, and attention level data. For example, based on the determine linguistic variables and membership functions that categorize the various parameters, the computing device may determine whether the driver is physically unable to operate the vehicle safely. If the operator is able to operate the vehicle safely, the process returns to 1208 to process the next image. If not, the process goes to 1232.

[00151] At 1232, the computing device may send a control signal to one or more vehicle systems to perform a function for operating the vehicle safely such as an automatic parking function, and obstacle avoidance function, a braking function or the like. For instance, the computing device may send signals to vehicle control systems to drive/park the vehicle autonomously.

[00152] The example processes described herein are only examples of processes provided for discussion purposes. Numerous other variations will be apparent to those of skill in the art in light of the disclosure herein. Further, while the disclosure herein sets forth several examples of suitable frameworks, architectures and environments for executing the processes, the implementations herein are not limited to the particular examples shown and discussed. Furthermore, this disclosure provides various example implementations, as described and as illustrated in the drawings. However, this disclosure is not limited to the implementations described and illustrated herein, but can extend to other implementations, as would be known or as would become known to those skilled in the art. [00153] Various instructions, methods, and techniques described herein may be considered in the general context of computer-executable instructions, such as computer programs and applications stored on computer-readable media and executed by the processor(s) herein. Generally, the terms program and application may be used interchangeably, and may include instructions, routines, modules, objects, components, data structures, executable code, etc., for performing particular tasks or implementing particular data types. These programs, applications, and the like, may be executed as native code or may be downloaded and executed, such as in a virtual machine or other just-in-time compilation execution environment. Typically, the functionality of the programs and applications may be combined or distributed as desired in various implementations. An implementation of these programs, applications, and techniques may be stored on computer storage media or transmitted across some form of communication media. [00154] Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. A method comprising: receiving, by one or more processors, from eyewear including one or more cameras, at least one image, the eyewear worn by an operator of a vehicle; determining, based at least in part on the at least one image, an operator field of view

(FOV); comparing the operator FOV with a recommended FOV ; and performing at least one action based on the comparing.

2. The method as recited in claim 1, wherein comparing the operator FOV with the recommended FOV comprises: receiving a plurality of consecutive images of the operator FOV captured over time; comparing individual images of the plurality of consecutive images with at least one recommended FOV to determine a difference; and performing the at least one action based on the difference exceeding a threshold difference for a threshold period of time.

3. The method as recited in claim 2, wherein the at least one action includes at least one of: sending a communication to cause information related to the recommended FOV to be displayed via the eyewear; sending a communication to cause a notification to be displayed via the eyewear; or sending a communication to cause information related to the recommended FOV to be presented on a display of the vehicle.

4. The method as recited in claim 1, further comprising: determining a location of the vehicle; and receiving the recommended FOV based at least in part on the location of the vehicle.

5. The method as recited in claim 1, wherein receiving, from the eyewear, the at least one image, comprises receiving an image including an eye of the operator, the method further comprising: determining at least one condition of the operator based at least in part on the image including the eye of the operator; and performing the at least one action or at least one additional action based on the at least one condition.

6. The method as recited in claim 5, wherein determining the at least one condition comprises determining an eye gaze of the operator based at least on identifying a plurality of key point features in the image including the eye and determining the eye gaze based at least in part on relative positions of the plurality of key point features.

7. The method as recited in claim 5, wherein determining the at least one condition comprises determining a heart rate of the operator based on comparing a plurality of the images of the eye.

8. The method as recited in claim 5, wherein determining the at least one condition comprises, based on comparing a plurality of the images of the eye, determining at least one of: a level of drowsiness of the operator; or a level of fatigue of the operator.

9. The method as recited in claim 1, further comprising: comparing the operator FOV with a calibration image; and determining a level of distraction of the operator based at least on comparing the operator FOV with the calibration image.

10. The method as recited in claim 1, further comprising: comparing the operator FOV with a calibration image; and determining a head pose of the operator based at least on comparing the operator FOV with the calibration image.

11. The method as recited in claim 1, further comprising: receiving the at least one image on a mobile device; performing at least a portion of image processing on the mobile device; receiving, by a vehicle computing device, the at least one image from the mobile device; and performing, by the vehicle computing device, the at least one action based on the comparing.

12. A system comprising: one or more computing devices configured by executable instructions to perform operations that include: receiving, from eyewear including one or more cameras, at least one image, the eyewear worn by an operator of a vehicle; determining, based at least in part on the at least one image, an operator field of view (FOV); comparing the operator FOV with a recommended FOV ; and performing at least one action based on the comparing.

13. The system as recited in claim 12, wherein receiving, from the eyewear, the at least one image, comprises receiving an image including an eye of the operator, the operations further comprising: determining at least one condition of the operator based at least in part on the image including the eye of the operator; and performing the at least one action or at least one additional action based on the at least one condition.

14. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of one or more computing devices, configure the one or more computing devices to perform operations comprising: receiving, from eyewear including one or more cameras, at least one image, the eyewear worn by an operator of a vehicle; determining, based at least in part on the at least one image, an operator field of view

15. The one or more non-transitory computer-readable media as recited in claim 14, wherein receiving, from the eyewear, the at least one image, comprises receiving an image including an eye of the operator, the operations further comprising: determining at least one condition of the operator based at least in part on the image including the eye of the operator; and performing the at least one action or at least one additional action based on the at least one condition.