EP4665272A2 - Orbital exenteration prothesis synchronized ocular movement integration system - Google Patents

Abstract

An orbital exenteration prosthesis system and method are disclosed that can synchronize with the ocular movement of the contralateral normal eye. The prosthesis device is configured to operate in the orbital cavity of a user in a comfortable manner, that is, without vibration or excess heat that can cause discomfort. The prosthesis device operates with a sensor system that is configured to determine the position of the contralateral normal eye and provide that information as a control signal to the orbital exenteration prosthesis located in the orbital cavity to match any synchronous dynamic actions of the contralateral normal eye. The sensor system beneficially employs a real-time AI model configured to operate in real-time control loop executing on a single board computer with constrained computing power that can provide an output matching the position and rate of change of the pupil of the contralateral normal eye.

Description

ORBITAL EXENTERATION PROTHESIS SYNCHRONIZED OCULAR

MOVEMENT INTEGRATION SYSTEM

RELATED APPLICATION

[0001] This PCT International Patent Application claims priority to, and the benefit of, U.S. Provisional Patent Application No. 63/484,867, filed February 14, 2023, which is incorporated by reference herein in its entirety.

BACKGROUND

[0002] Orbital exenteration is a radical surgical procedure in which the eyelids, eye, and orbital contents, including extraocular muscles, optic nerve, fat, and lacrimal gland, are removed en bloc. This procedure is commonly performed to treat malignant periocular tumors invading the orbit, intraocular tumors with extraocular extension, or primary and secondary orbital malignancies. The surgical aim is to achieve tumor-free margins and is also performed in painful or life-threatening orbital infections or inflammations. The post-surgical outcome is an empty orbital cavity.

[0003] Loss or absence of an eye and orbital contents is physically and psychologically traumatizing and can severely affect human interactions. The psychological and cosmetic rehabilitation begins shortly after socket healing with the fitting of an orbital exenteration prosthesis to approximate the form of the contralateral normal side. The fabrication of an exenteration prosthesis is an arduous and time-intensive process. A cosmetic exenteration prosthesis consisting of non-blinking eyelids and an acrylic ocular prosthesis insert is fabricated to match the shape and color of the contralateral normal eye. The exenteration prosthesis is affixed to the entrance of the empty socket with glue, double-sided tape, or a magnet.

[0004] The principal drawback of the conventional “gold standard” orbital exenteration prosthesis, even if the cosmetic match is perfect, is the inability to match any synchronous dynamic actions of the contralateral normal eye - eyelid blinking and conjugate eye movements. As a result, the prosthesis is immediately and easily noticeable once the patient blinks or moves the normal eye. The prosthetic eye/eyelid remains stationary or adynamic. This discrepancy is often noticed by observers immediately and is a source of much consternation for the wearer. Psychologically, the wearer still does not feel whole.

[0005] There is a benefit to improving orbital exenteration prosthesis. SUMMARY

[0006] An orbital exenteration prosthesis system and method are disclosed that can synchronize with the ocular movement of the contralateral normal eye. The prosthesis device is configured to operate in the orbital cavity of a user in a comfortable manner, that is, without vibration or excess heat that can cause discomfort. The prosthesis device operates with a sensor system that is configured to determine the position of the contralateral normal eye and provide that information as a control signal to the orbital exenteration prosthesis located in the orbital cavity to match any synchronous dynamic actions of the contralateral normal eye. The sensor system beneficially employs a real-time Al model configured to operate in a real-time control loop executing on a single board computer with constrained computing power that can provide an output matching the position and rate of change of the pupil of the contralateral normal eye.

[0007] The mechanical design of the orbital exenteration prosthesis system can be implemented as an orbital eye robot device that can be used as a part of a robotic system that can mimic human movement.

[0008] A mobility prosthesis may move synchronously with the normal eye and may be made to blink.

[0009] In an aspect, a system is disclosed comprising a prosthetic eye system configured to be placed in an orbital cavity of a person, the prosthetic eye system comprising a motorized eyeball structure configured to move in two or more directions; a sensor system comprising a sensor configured to acquire an image of an eye contralaterally located to the prosthetic eye system; and a controller, the controller comprising: a processor; a memory having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to: acquire an image of the eye contralateral of the prosthetic eye system; determine, via a real-time Al model, a position of a pupil of the eye; and output a command to the prosthetic eye system to move the motorized eyeball structure in a manner corresponding to the eye contralaterally located to the prosthetic eye system.

[0010] In some embodiments, the prosthetic eye system comprises a first motor unit configured to move in a first direction; and a second motor unit operatively coupled to the first motor unit to move in a second direction, wherein the second direction is perpendicular to the first direction.

[0011] In some embodiments, the first motor unit and the second motor unit each comprises a DC motor. [0012] In some embodiments, the sensor system and controller are each located in an eyeglass frame configured to be worn by the person.

[0013] In some embodiments, the controller is implemented in a single-board computer that is embedded into an ipsilateral temple frame of the eyeglass frame.

[0014] In some embodiments, the sensor system comprises an IR camera that is housed at an inferior temporal quadrant of an eyeglass frame.

[0015] In some embodiments, the real-time Al model comprises a residual CNN.

[0016] In some embodiments, the residual CNN includes residual connections and at least one of Squeeze and Excitation (SE) attention, Atrous Spatial Pyramid Pooling (ASPP), and a combination thereof.

[0017] In some embodiments, the residual CNN was trained by quantizing the model parameters to speed up the predictions using the Quantization Aware Training (QAT) operation.

[0018] In some embodiments, the real-time Al model is configured to execute in realtime time steps of greater than 30 ms per classification on a resource-constrained singleboard computer.

[0019] In another aspect, a prosthetic eye system is disclosed that is configured to be placed in an orbital cavity of a person (or a robot system), the prosthetic eye system comprising a motorized eyeball structure configured to move in two or more directions comprising: a first motor unit configured to move in a first direction; and a second motor unit operatively coupled to the first motor unit to move in a second direction, wherein the second direction is perpendicular to the first direction.

[0020] In some embodiments, the first motor unit and the second motor unit each comprises a DC motor.

[0021] In another aspect, an eyeglass frame apparatus is disclosed that is configured to be worn by a person, the apparatus comprising: a first ipsilateral temple frame portion; a lens region having an inferior temporal quadrant; a sensor embedded in the inferior temporal quadrant; and a computing device embedded in the ipsilateral temple frame portion and operatively connected to the sensor, the computing device having a controller having a processor; a memory having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to: acquire an image from the sensor; execute a real-time Al model in a real-time loop to determine a position of a pupil of a non-prosthetic eye in the image; and output a command to a prosthetic eye system to move a motorized eyeball structure in a manner corresponding to the non-prosthetic eye.

[0022] In some embodiments, the apparatus further includes a second ipsilateral temple frame portion that is operatively coupled (e.g., hingeably coupled) to the lens region; and a rechargeable energy storage unit embedded within the second ipsilateral temple frame portion.

[0023] In another aspect, a non-transitory computer-readable medium is disclosed having instructions stored thereon for real-time detecting of a pupil location in an image of an eye, wherein execution for the instructions by a processor, causes the processor to: obtain an image of the eye contralaterally located to the prosthetic eye system; determine, via a realtime Al model executing in a real-time control loop, a position of a pupil of the eye; and output a command to a prosthetic eye system to move the motorized eyeball structure in a manner corresponding to the eye contralaterally located to the prosthetic eye system.

[0024] In some embodiments, the real-time Al model comprises a residual CNN.

[0025] In some embodiments, the residual CNN includes residual connections and at least one of Squeeze and Excitation (SE) attention, Atrous Spatial Pyramid Pooling (ASPP), and a combination thereof.

[0026] In some embodiments, the residual CNN was trained by quantizing the model parameters to speed up the predictions using the Quantization Aware Training (QAT) operation.

[0027] In some embodiments, the real-time Al model is configured to execute in realtime time steps of greater than 30 ms per classification on a resource-constrained portable computer.

[0028] In another aspect, a non-transitory computer-readable medium is disclosed having instructions stored thereon for real-time detection of an eyelid location in an image of an eye, wherein execution for the instructions by a processor, causes the processor to: obtain an image of the eye contralaterally located to the prosthetic eye system; determine, via a realtime Al model executing in a real-time control loop, a position of an eye lid of the eye; and output a command to a prosthetic eye system to move a motorized eyelid in a manner corresponding to the eyelid of the eye contralaterally located to the prosthetic eye system.

BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments and, together with the description, serve to explain the principles of the methods and systems. The patent or application file contains at least one drawing executed in color.

[0030] Fig. 1 shows an orbital exenteration prosthesis system comprising an orbital exenteration prosthesis device and pupil tracking system that is configured to synchronize with ocular movement or pupil of the contralateral normal eye, in accordance with an illustrative embodiment.

[0031] Fig. 2A includes a plurality of images showing various components of an example implementation of the exenteration prosthesis system, in accordance with an illustrative embodiment.

[0032] Fig. 2B shows an example implementation of the orbital exenteration prosthesis device, in accordance with an illustrative embodiment.

[0033] Fig. 2C shows a set of images of the fabricated prototyped motorized unit of Fig. 2A from the front view, in accordance with an illustrative embodiment.

[0034] Fig. 2D shows an example motor unit that may be used in the motorized assembly or unit, in accordance with an illustrative embodiment.

[0035] Figs. 2E and 2F show an example implementation of the orbital exenteration prosthesis device having a blinking mechanism, in accordance with an illustrative embodiment.

[0036] Figs. 2G and 2H show the elements of the blinking mechanism of Figs. 2E and 2F.

[0037] Fig. 3 A shows the range of operation of the orbital exenteration prosthesis.

[0038] Fig. 3B is a diagram of the physiology of the human ocular system.

[0039] Fig. 4 shows a method 400 to operate an orbital exenteration prosthesis system in accordance with an illustrative embodiment.

[0040] Figs. 5A, 5B, 5C, and 5D show various aspects of a real-time Al model that may be executed in a real-time loop to detect the position of an eye contralaterally located to the orbital exenteration prosthesis system, in accordance with an illustrative embodiment.

[0041] Figs. 6A, 6B, 6C, 6D, 6E, 6F, 6G, and 6H show experimental results of a prototyped orbital exenteration prosthesis system, in accordance with an illustrative embodiment.

[0042] Figs. 7A, 7B, 7C, 7D, 7E, 7F, 7G, and 7H who the workflow steps for fabrication of an exenteration prosthesis, according to one illustrative embodiment. DETAILED DESCRIPTION

[0043] Each and every feature described herein, and each and every combination of two or more of such features, is included within the scope of the present invention, provided that the features included in such a combination are not mutually inconsistent.

[0044] Example System

[0045] Fig. 1 shows an orbital exenteration prosthesis system 100 comprising an orbital exenteration prosthesis device 102 and pupil tracking system 104 that is configured to synchronize with ocular movement or pupil 106 of the contralateral normal eye 108. The prosthesis device 102 is configured to operate in the orbital cavity 110 of a user in a comfortable manner, that is, without vibration or excess heat that can cause discomfort and that can match the synchronous dynamic actions of the contralateral normal eye 108. The prosthesis device 102 operates with a pupil tracking system 104 comprising a sensor 112 (shown as “IR Camera” 112’) configured to determine the position of the pupil 106 of the contralateral normal eye 108 (the normal left or right eye) and provide that information as a control signal 114 (shown as a wireless signal 114) to the orbital exenteration prosthesis device 102 located in the orbital cavity 110 to drive movement of the prosthesis eye 116 that matches the synchronous dynamic actions of the contralateral normal eye 108. The prosthesis eye 116 includes an iris portion and a pupil portion (collectively shown as 118), e.g., formed as an acrylic eye prosthesis. IR sensors can operate in low light and daylight conditions. Other sensor types can be used, e.g., photodiodes, CCDs, etc.

[0046] The pupil tracking system 104, as the sensor system, beneficially employs an Al model 120 configured to operate in real-time (e.g., greater than 30 ms control resolution) on portable computing device 122 (shown as a single board computer 122’) that can provide an output matching the position and rate of change of the pupil of the contralateral normal eye. The single board computer 122’ includes a wireless network interface 126 (shown as “Interface” 126) to communicate with the prosthesis device 102.

[0047] In the example shown in Fig. 1, the pupil tracking system 104 is implemented via a single board computer 122’ with constrained computing power that can be integrated into a wearable device, e.g., eyewear. The frame 128 of the eyewear can be customized to be contemporaneous in size of the external prosthesis 124 to mask any boundaries of the orbital exenteration prosthesis device 102 as it seats in the orbital cavity 110.

[0048] Fig. 2A includes a plurality of images showing various components of an example implementation of the exenteration prosthesis system 100 (shown as 100’). The image 202 shows the mechanical design of the orbital exenteration prosthesis system 100’, including the orbital exenteration prosthesis device 102 (shown as 102’). The motorized unit of the orbital exenteration prosthesis device 102’ is shown, and it is fabricated by a 3D manufactured part with integrated motors. Image 204 shows the same orbital exenteration prosthesis system 100’ placed in a customized rubber mold prosthesis 124 (shown as 124’). Image 206 shows the orbital exenteration prosthesis device 102’ and mold prosthesis placed in an orbital cavity of a replicate skull. Image 208 shows two example frames (128a, 128b) of the tracking system 104. Image 210 shows the frame having an integrated computing device 122 (shown as 122’) built into one of the temples 212 of the eyewear. The other temple is configured with an energy storage device (e.g., rechargeable batteries) sized to operate in continuous operation for a pre-defined period of time.

[0049] Referring back to Fig. 1, to capture the pupil movement of the healthy eye in realtime, the pupil tracking system 104 employs a sensor 112 to determine the position of the pupil 106 of the contralateral normal eye 108. In some embodiments, the sensor 112 is an infrared 200 Hx monocular camera. The sensor 112 is preferably mounted on the inferior temporal quadrant of eyeglass frame 128, though may be mounted at other quadrant of the frame. The sensor 112 may capture an image of the pupil and send the images to the embedded camera PCB, e.g., on the ipsilateral temple frame. A machine learning-based pupil detection algorithm may be employed, using integrated video processing software, to convert eye coordinates into prosthesis positions. The algorithm may operate in real time with the best trade-off model that is augmented by automated calibration. A motor control algorithm may be implemented to convert the eye position into smooth conjugate saccade prosthesis movement. The output signal may be provided by wire or wireless transmission of signals to a miniature motor attached to the back surface of an exenteration prosthesis. The miniature motor unit with the attached exenteration prosthesis may be positioned in a sclera-colored sphere and coupled to the posterior cavity of a silicone exenteration prosthesis (e.g., 3-D printed silicone exenteration prosthesis).

[0050] Example Eyeglass Frame. In the example shown in Fig. 2A, the inferotemporal quadrant of the eyeglass frame of the seeing eye preferably houses the camera with infrared lighting and sensor. The PCB/microcontroller for the camera may be embedded in the ipsilateral temple arm of the frame of the camera/sensor side. The battery to power the motor unit and a computer processor with wiring integration may be embedded in the temple arm of the frame on the same side as the DC motors. The frame design may be customized and, e.g., 3-D printed for each patient to ensure a comfortable fitting.

[0051] Example Orbital Exenteration Prothesis Device

[0052] Fig. 1 shows an example of the orbital exenteration prosthesis device 102. Fig. 2B shows an example implementation of the orbital exenteration prosthesis device 102 (shown as 102’). The device (e.g., 102, 102’) can be fabricated to be compact, lightweight, and low-cost. Additionally, the device (e.g., 102, 102’) can be configured, preferably using a DC motor to operate without generating excess heat as the device seats in the orbital cavity 110 and without vibration that can cause discomfort for the user. Other moto types may be used, including those described herein, with additional customization.

[0053] DC motors are preferable because of the straightforward control operation, low power consumption, and high speed of action. In the example shown in Fig. 2A, the motorized assembly or unit 134 includes planetary gearboxes integrated DC motors that can control the prosthesis movement in vertical and horizontal directions. Torsional springs may be used within the mounts of each motor to revert the motion to the eye center when the current is not applied. The configuration of Fig. 2A can simplify the calibration and minimize the positioning errors after extensive prosthesis usage.

[0054] Motorized Assembly or Unit Design. In the example shown in Fig. 1, the orbital exenteration prosthesis device 102 includes two motorized assemblies configured for two- dimensional motion (e.g., x and y) that can move synchronously, via a pupil tracking program, with the healthy eye of the use. The orbital exenteration prosthesis device 102 includes a first motor 130 configured to rotate (132) the prosthesis assembly 134 in the first direction, e.g., in an x-y plane. The first motor 130 is operatively connected to a hinge located in the prosthesis assembly 134. The prosthesis assembly 134 includes a first gear 136 (e.g., located in the y-z direction) that is operatively coupled to a second gear 138 that is driven by a second motor 140. The second motor 140 and gears 136, 138 are located on the prosthesis assembly 134 and can be rotated by the first motor 130. The second motor 140 can rotate the assembly in the z-plane to move the pupil up and down. The first motor 130 may be located at a bottom portion of the prosthesis device 102 to provide weighing at that region of the orbital cavity. As shown in Fig. 1, the battery unit 142 for the prosthesis device 102 and the controller 144 and network interface 146 may be similarly located at the bottom portion of the prosthesis device 102. The centrally located masses of the first motor 130, the battery unit 142, and other elements within the orbital cavity help to restore the normal anatomical relationship with the contralateral normal eye to optimize congruous eye movements.

[0055] To make the prosthesis device 102 inconspicuous or so to an observer, the device 102’ is configured to operate with low lag time. The orbital exenteration prosthesis device (e.g., 102, 102’), as a compact mechanical device, can mimic the natural motion of a healthy eye for patients with orbital exenteration. The device is smaller than the orbital cavity size, e.g., <25 mm, for the average adult. The system is low weight and is configured for fast response, e.g., 20-25 ms response time. To this end, the device 102 is inconspicuous at static and dynamic states.

[0056] The system is robust for all users and able to compensate for movement, aided in part by the centrally located masses of the motorized assemblies within the orbital exenteration prosthesis device 102. The system is integrated on an eyeglass frame and employs PCB for the camera and computer. The device 102 is also configured for comfort and/or safety for the user. The system is minimized in weight for the comfort of wear. The system operates with low or no vibration, which could otherwise induce headaches. The system is sealed and insulated to prevent electric shocks. Response time is critical to be inconspicuous to a nearby observer.

[0057] Additional Motorized Assembly or Unit Design. Fig. 2B shows another example motorized assembly or unit 134 (shown as 134’) of the orbital exenteration prosthesis device (shown as 102”). The motorized assembly or unit 134’ includes the first motor 130 configured to rotate (132) the prosthesis assembly 134’ in the first direction, e.g., in the x-y plane. The first motor 130 is operatively connected to a hinge structure 214 formed by the prosthesis assembly 134’. The prosthesis assembly 134’ includes a second hinge structure 216 that is driven by a second motor 140. The second motor 140 and the hinge structures 214, 216 are located on the prosthesis assembly 134’ and can be rotated by the first motor 130. The second motor 140 can rotate the assembly 134’ in the z-plane to move the pupil up and down. The second hinge structure 216 includes a first end 218 that connects to the eyeball structure 220 at the iris and pupil region 118 of the eye and a second end to retain the prosthesis assembly 134’ with the eyeball structure.

[0058] Prototyped motorized assembly or unit. Fig. 2C shows a set of images of the fabricated prototyped motorized unit of Fig. 2A from the front view. The prototyped motorized unit is shown with a prosthetic pupil cover that is moved by the motor to different ranges of motion. [0059] Yet Additional Motorized Assembly or Unit Design. Fig. 2D shows an example motor unit that may be used in the motorized assembly or unit (e.g., 134). The motor unit may be based on a stepper motor or linear motor. Consideration must be given to heat dissipation for the comfort of the user while the prosthesis is placed in the orbital cavity. The system could be customized to improve the matching speed and/or miniaturized to fit the orbital cavity.

[0060] Blinking Mechanism. Figs. 2E-2H show an example implementation of a blinking mechanism compatible with the orbital exenteration prosthesis system, e.g., comprising the orbital exenteration prosthesis device 102 and pupil tracking system 104). The blinking mechanism 250 provides another element of realism to orbital exenteration prosthesis system by tracking the other eye and initiating the blinking of the orbital exenteration prosthesis device (shown as 102”) in view of measurement of the biological eye.

[0061] The blinking mechanism 250 includes a blinking motor 252 additional to the eye ball movement motors, shown as motors 130, 140 (shown as 130’ and 140’). Motors 130’, 140’ provide actuation for movement of the prosthesis in each of a first direction (e.g., in an x-y plane to move the pupil side to side) and a second direction (e.g., in the z-plane to move the pupil up and down), while motor 252 of the blinking mechanism 250 controls the up and down movement of an eyelid 254 that covers and/or exposes the front of the prosthesis (e.g., the iris and pupil). The motors 130’, 140’, 254 are sufficiently sized to collectively fit entirely within the posterior bulbous chamber behind the exenteration prosthesis of the system. The motors 130’, 140’, 254 have sufficient power to accomplish fast movements while avoiding excess heat dissipation into the orbital cavity. In one example, the motors are each 3 V 2-gear DC motors.

[0062] The blinking mechanism 250 includes a bevel gear 256 coupled to the motor 252. The bevel gear 256 is operatively positioned and coupled to a custom contoured wedge gear 258. Additionally, the eyelid 254 includes a shaft 262 on one end towards the side of the eye. The wedge gear 258 defines an opening 260 through which the shaft 262 of the eyelid 254 extends and is coupled.

[0063] In operation, the motor 252 causes the bevel gear 256 to rotate, initiating a corresponding rotation of the wedge gear 258. In some implementations, a gear reduction or other type of gear system may be implemented (e.g., for fast motion or low torque operations). The wedge gear 258 causes the eyelid 254 to rotate around the outside of the prosthesis within the orbital cavity. The eyelid 254 then covers the front of the eye (e.g., the iris and pupil). The eyelid 254 may cover the front of the eye for only a brief period of time, corresponding to the length of time for a human to blink. In other implementations, the eyelid may stay over the front of the eye to match the motion of the seeing eye (e.g., closing one’s eyes).

[0064] The eyelid 254 is a curvilinear plastic piece configured to rotate in place about the orbital cavity. However, in other implementations, the eyelid is a soft plastic piece (e.g., silicone or flexible polymer sheet) or other soft material mimicking human skin texture and color.

[0065] In use, the pupil tracking system 104 can also track the eyelid of the seeing eye or the pupil to send signals to the blinking mechanism 250 to perform a blinking operation. For example, the pupil tracking system 104 employs a sensor 112 to determine the position of the pupil 106 of the contralateral normal eye 108 may use the same signal to for the blinking mechanism 250. In some implementations, the pupil tracking system 104, and the associated Al model 120, monitors the pupil to determine if the pupil is no longer visible. Once the pupil is no longer visible, a blinking operation is initiated to move the eyelid 254 over the pupil of the prosthesis and back again. In other implementations, the pupil tracking system 104, and the associated Al model 120, monitors the eyelid margin of the seeing eye to determine the exact position in the upward/downward direction of the eyelid of the seeing eye. Then, the blinking mechanism 250 moves in conjunction with the eyelid margin of the seeing eye.

[0066] Prothesis Motion Operation. Fig. 3 A shows the range of operation of the orbital exenteration prosthesis 102. The prosthesis 102 is configured for dual motion based on 2D vectors. In the x-axis, the prosthesis 102 is configured, via a first motor (e.g., at the bottom location), to move ±44 degrees in the left and right direction, and in the y-axis, the prosthesis 102 is configured via a second motor (e.g., at a top location) to move ±28 degrees in the up and down direction. Fig. 3B is a diagram of the physiology of the human ocular system. [0067] Example Methods

[0068] Fig. 4 shows a method 400 to operate an orbital exenteration prosthesis system (e.g., 100, 100’) in accordance with an illustrative embodiment. Method 400 includes placing 402 the prosthetic eye system (e.g., 102, 102’, 102”) in an orbital cavity of a person, the prosthetic eye system comprising a motorized eyeball structure configured to move in two or more directions. In some embodiments, the prosthetic eye system comprises a first motor unit configured to move in a first direction; and a second motor unit operatively coupled to the first motor unit to move in a second direction, wherein the second direction is perpendicular to the first direction. In some embodiments, the first motor unit and the second motor unit each comprises a DC motor.

[0069] Method 400 includes acquiring (404) an image (e.g., 502) of the eye contralaterally located to the prosthetic eye system (e.g., 102, 102’, 102”). In some embodiments, the sensor system and controller are each located in an eyeglass frame configured to be worn by the person. In some embodiments, the controller is implemented in a single-board computer that is embedded into an ipsilateral temple frame of the eyeglass frame. In some embodiments, the sensor system comprises an IR camera that is housed at an inferior temporal quadrant of eyeglass frame.

[0070] Method 400 includes determining (406), via a real-time Al model executing in a real-time control loop, a position of a pupil of the eye. In some embodiments, the real-time Al model comprises a residual convolutional neural network (CNN). In some embodiments, the residual CNN includes residual connections and at least one of Squeeze and Excitation (SE) attention, Atrous Spatial Pyramid Pooling (ASPP), and a combination thereof. In some embodiments, the residual CNN was trained by quantizing the model parameters to speed up the predictions using Quantization Aware Training (QAT) operation.

[0071] Method 400 includes outputting (408) a command to the prosthetic eye system to move the motorized eyeball structure in a manner corresponding to the eye contralaterally located to the prosthetic eye system. In some embodiments, the real-time Al model is configured to execute in real-time time steps of greater than 30 ms per classification on a resource-constrained single board computer.

[0072] Real-Time Al Model for Orbital Exenteration Prosthesis System

[0073] The pupil tracking system 104, as the sensor system, beneficially employ an Al model 120 configured to operate in real-time (e.g., greater to 30 ms control resolution) on portable computing device 122 (shown as a single board computer 122’) that can provide an output matching the position and rate of change of the pupil of the contralateral normal eye. The same, or similar, tracking system can be employed to sense and actuate the blinking mechanism, in which the onset of a blink in the healthy eye initiates a blink in the prosthesis. [0074] The Al model 120 may be implemented using a CNN-based model, e.g., Residual CNN (also referred to as Res-CNN), that employs residual connections, Squeeze and Excitation (SE) attention, and Atrous Spatial Pyramid Pooling (ASPP) to improve the prediction performance without complicating the model. The model 120 may employ transfer learning by training synthetic images, then fine-tuning them with authentic eye images. The model 120 may be trained by fully quantizing the model parameters to speed up the predictions using the Quantization Aware Training (QAT) strategy to obtain accurate predictions. Experiments show that the quantized Res-CNN model trained by QAT strategy with 40x30 resolution images can provide robust yet real-time predictions with an average of 1.351 Root Mean Square Error (RMSE) for the pupil center predictions and 8.317 ms response time per image on a single-board computer, Raspberry Pi.

[0075] Existing deep-fully convolutional network (DFCN) based pupil segmentation models have been shown to perform robust and accurate pupil detection. However, such models require high computational resources for real-time predictions, which may not be suitable in their current form for low computational, wearable devices executing on computing-power-constrained single board computers such as Raspberry Pi. Simple FCN models, on the other hand, can provide real-time but not the accuracy performance required for the application.

[0076] Fig. 5A shows an example Al model 120 (shown as 120’) configured to operate in real-time (e.g., greater to 30-millisecond control resolution) on portable computing device 122 (shown as a single board computer 122’) that can provide an output matching the position and rate of change of the pupil of the contralateral normal eye. In the example shown in Fig. 5A, input images 501 acquired by sensor 112 may be fed into the model 120’ without pre-processing. In the QAT training strategy, the Al model 120’ was trained by converting the image tensors from floating points to quantized points. Then the lightweight, robust Residual CNN model with fully quantized layers may be trained using QAT, and pupil center coordinates are directly obtained. The output of the Al model 120’ is provided to motor control 503 that provide the control signals to a network transmitter 505 to direct control of the prosthesis device (e.g., 102, 102’).

[0077] Fig. 5B shows example input images 501 acquired by sensor 112.

[0078] Fig. 5C shows an example Residual CNN model architecture 500 (previously referenced as Al model 120) that include three residual CNN blocks 502 (shown as 502a, 502b, 502c). While other block configuration may be used, it is noted that 2 blocks was observed to not provide sufficient accuracy without additional modifications and 4 or more blocks employs more computing resources than available on the tested hardware.

[0079] In the example shown in Fig. 5C, the first block 502a includes a convolutional layer 504 configured with 7^7 kernel, Batch Normalization [31], and ReLu [32] nonlinearity [10, 21, 31], Another convolutional block follows the first conventional block configured with a 3 *3 kernel and a stride of 2 without any nonlinearity. The output of the two convolutional block sequences is summed with another 3x3 convolutional layer with a stride of 2 with Batch Normalization, which is applied as a skip connection. These skip connections may overcome the vanishing gradient problem in deep CNNs and significantly increase performance in computer vision applications [10],

[0080] The same procedure is also followed for the second and third main blocks 502b, 502c. For the first, the second, and the third main blocks (502a, 502b, 502c), the model 500 includes 32, 64, and 128 filters for all convolutional layers, respectively. Except for the first convolutional layer, which has a 7x7 kernel size, all the convolutional layers in each main block employed a 3x3 kernel size. After the first and the second main blocks (502a, 502b), the model 500 employed a pioneer attention module, namely the SE network [28] (shown as 506 in Fig. 5D).

[0081] At the end of the third main block (502c), the module 500 includes an ASPP [11] (504) that is utilized to capture the long-range dependencies of the feature maps before the fully connected layers. Subsequently, the model 500 includes a Global Average Pooling (GAP) that is used in the feature maps to achieve a fixed dimension for the fully connected layers. The GAP operation also helps the model to collect the global information for each feature map and reduces the computational cost. Finally, the module 500 includes a linear- batch normalization-ReLu sequence 508 with 64 neurons that can be used for the regression operation. The module 500 includes a single linear with two neurons with a sigmoid activation function (510) to provide the output predictions 512 comprising an x-y position.

[0082] Fig. 5D shows the SE network architecture 506. In the example shown in Fig. 5C, the SE Networks 506 includes a squeeze block that collects the global spatial information by applying GAP (508). Then, the excitation block captures these channel-wise relationships and produces an output attention vector using two fully connected layers with ReLu non-linearity (510). Finally, these attention vectors give weights to each input feature by multiplying attention vectors with the original input feature maps [33],

[0083] Fig. 5D also shows the Atrous Spatial Pyramid Pooling (ASPP) [11] (504). The ASPP 504 appears at the end of the third main block and may be utilized to capture the long- range dependencies of the feature maps before the fully connected layers. ASPP operation 504 may be motivated by the success of Spatial Pyramid Pooling (SPP) [29, 34] and is widely used in computer vision applications [35-38], The first part of the ASPP variation in Fig. 5D includes three convolutional blocks 512 (each following a convolutional operation-batch normalization-ReLu nonlinearity sequence) with 3^3 kernel size and 6, 12, and 18 dilation rates, respectively.

[0084] Convolutional operation with different dilation rates may capture the long-range dependencies among different pixels to provide a better feature extraction performance. The padding rates for each convolutional block may have values of 6, 12, and 18, respectively, to keep the original image resolution. The feature maps may obtain from those three convolutional blocks with different dilation rates are then concatenated 514 along the channel dimension. A final convolutional operation 516 configured with a 1 ^X 1 kernel may be performed to achieve the final feature maps. GAP may be used on the feature maps obtained by the ASPP block to achieve a fixed dimension for the fully connected layers. Such an operation also may help the model to collect the global information for each feature map and reduces the computational cost. Finally, a linear block that also follows a linear-batch normalization-ReLu sequence with 64 neurons may be used for the regression part. Then the output predictions are achieved using a single linear with two neurons with a sigmoid activation function.

[0085] Experimental Results and Examples

[0086] A study was conducted to develop an Al model to detect/track pupil location and direct movement of the prosthesis in an orbital exenteration prosthesis system. The study included operations such as residual connections, SE, and ASPP to improve the prediction performance without significantly increasing the model parameters. The study also leveraged transfer learning by initially training the model with a dataset that only includes synthetic eye images and fine-tuning the same model with the actual eye dataset. Furthermore, the study integrated the QAT strategy to increase the speed of the predictions.

[0087] Experiments showed that the quantized Res-CNN model trained with QAT strategy using 40x30 resolution images performed well on the LPW test dataset while providing real-time predictions on Raspberry Pi with an average of 8.317 ms response time per image. It was also possible to consider 60x45 input images when real-time prediction performance is of interest. The quantized Res-CNN models with QAT outperformed conventional Res-CNN models with almost the same prediction accuracy.

[0088] Datasets. This study used the LPW [20] dataset to train our model (Fig. 6A). The characteristic of the LPW dataset includes varying conditions such as gender, nationality, environment (indoor or outdoor), lighting type (natural or artificial), and makeup condition. The LPW dataset employed in the study included 66 high-resolution (640x480) videos obtained from 22 different participants. For each patient, three videos in different conditions are accepted, and each video consists of 2000 frames recorded at 95 FPS (nearly 130,856 frames). In the study, instead of all 130,856 frames, the study used 40 frames per video since (2640 in total) images obtained from high FPS will result in an increased number of similar images, which can affect the training performance.

[0089] The study used, as for the transfer learning dataset, the SynthesEyes Dataset [30], which includes a collection of dynamic eye regions built with computer graphics. The dataset included 11,382 synthetic eye images with varying conditions, including skin color, skin smoothness, and eye shape (Fig. 6B). More details regarding the dataset can be found in [30],

[0090] Training Procedure. Before training, the study converted the input images for both SynthesEyes and LPW datasets to grayscale and resized from 640x480 to 40x30 using bilinear interpolation. Although resizing images before training causes information loss and may sacrifice prediction accuracy, the main aim of this study was to perform a robust CNN model that can perform real-time predictions. Therefore, images were resized to a fixed resolution of 40x30 to obtain real-time predictions. The dataset was randomly divided into training and validation groups with a ratio of 8:2. That is, 80% of the data was used for training, and the remaining 20% was used for validation. Adam [39] optimization method with a 10'⁵ initial learning rate in the training procedure was used to update the model parameters. Mean Square Error (MSE) was the loss function since the study aimed to find the pupil center coordinates directly. The formulation of MSE is provided in Equation 1.

[0091] In Equation 1 , TV is the total training data points, y_t is the target value, and y_t is the model prediction. As for the performance metrics, Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) were used. The formulations for MAE and RMSE metrics are provided in Eqs. (2) and (3), respectively. RMSE = (3)

[0092] For the LPW dataset, the instant Res-CNN model of the study was trained for 5,000 epochs whereas, for the SynthesEyes dataset, the model was trained for 600 epochs. The batch size for both datasets for training was selected as 32. Several different transforming methods, including random shifts and rotations (-60, 60 degrees), horizontal and vertical flips, gaussian and glass blurs, and gaussian noise, were applied to the training input to improve the validation performance images. The images were only resized for the validation dataset and converted to grayscale.

[0093] Testing Procedure. The study used a single-board computer to test the Res-CNN model. The study imported the model to a Raspberry Pi 4B with a camera connected to its MIPI CSI port. Table 1 shows the specifications of the Raspberry Pi unit.

Table 1

[0094] The study ran the model on C++ by changing the processor affinity manually to utilize all 4 cores of the CPU. The study loaded the model into the program, and the video stream was started at 640x480 pixels and 35 FPS, enabling a new input frame every 28.57 ms. The input image was resized to 40x30 and was converted to grayscale to correspond to the format of the input layer of the model. The input tensor was normalized from 0-255 to 0- 1, and a forward pass was performed on the model. The study calculated the time it took the Raspberry Pi to execute the forward pass and recorded the results for each frame. The study computed the average response time of 1000 frames.

[0095] Results

[0096] Model Performance Evaluation on the Datasets. Fig. 6C shows the RMSE curves during training for the initial quantized Res-CNN model. The training and test data learning curves show that the proposed model can learn the pupil center coordinates from the synthetic eye images. The best training and the test RMSE values were achieved as 1.654 and 1.091, respectively. These values correspond to coordinate error. Image resolution after resizing the training procedure was 40x30, which can also be considered coordinates.

[0097] Fig. 6D demonstrates the representative pupil center predictions obtained from the test dataset. In these images, the green dot represented the prediction by the Res-CNN model, whereas the red dot identified the correct center of the pupil. According to the predictions in Fig. 6D, the initial Res-CNN model can accurately detect the pupil center with nearly a 1 -pixel error on average.

[0098] The aim of pre-training the Res-CNN model on synthetic eye images was to enhance the prediction performance on the authentic eye images by initializing a new model with its learned parameters. This may improve the prediction performance for real eye images since the initial Res-CNN model can provide prior information on extracting essential features by training with synthetic eye images.

[0099] It can be observed in Fig. 6D that the initial Res-CNN model can provide a generalizable performance for synthetic eye images. Thus, it is possible to transfer its trained parameters to initialize a new model with the same architecture, which can then be fine-tuned on the authentic eye images.

[0100] Fig. 6E shows the effect of transfer learning on the LPW dataset. Specifically, Fig. 6E shows the MAE and RMSE comparison for the models trained with and without transfer learning. In Fig. 6E, the ResCNN corresponds to the model trained from scratch, whereas ResCNN-TL corresponds to the model with a transfer learning strategy. The weight initialization strategy recommended by [40] was used for the Res-CNN model that was trained from scratch.

[0101] It can be seen in Fig. 6E that the transfer learning framework improved the detection performance. For the model trained from scratch, the RMSE and MAE values for the pupil center predictions were achieved as 1.447 and 0.997, respectively. Whereas for the transfer learning model, the RMSE and MAE values were committed as 1.351 and 0.879, respectively. To visualize the improved prediction performance of the transfer learning model, some pupil center predictions on the LPW test dataset are provided in Fig. 6F.

[0102] Fig. 6F shows the pupil center predictions using the transfer learning model on the LPW test dataset. According to the projections in the figure, the Res-CNN model with the QAT strategy can produce accurate pupil center predictions. In the literature, it is generally assumed that one can interpret the pupil center prediction as true positive only if the Euclidian distance between the prediction and the ground truth coordinates is less or equal to 5 pixels [17, 19, 20], Since the mean RMSE value for the test dataset is 1.351, it was possible to conclude that the model developed in the study provided accurate and reliable predictions. [0103] Prediction Speed of Res-CNN on Raspberry Pi. Since another purpose of implementing the QAT strategy was to obtain accurate predictions using small and portable devices at high speeds, the study conducted real-time tests using the quantized models on Raspberry Pi.

[0104] Fig. 6G shows the real-time prediction response times using quantized and nonquantized Res-CNN models at various input image resolutions. To obtain consistent response time measurements, the mean, standard deviation, maximum, and minimum response time values were calculated among 1000 sequential real-time predictions. In Fig. 6G, for each input image resolution, it can be observed that the quantized Res-CNN models significantly reduced the prediction response times. The quantized Res-CNN with 40x30 input image resolution achieved an average of 8.317 ms response time, whereas the nonquantized Res-CNN model responded at 91.90 ms. Such a difference in response times was, in fact, a direct consequence of the quantization strategy, which reduced the complexity of the model by converting the floating-point model parameters to integers hence reducing the complexity of the mathematical operations.

[0105] It may still be convenient to use the quantized Res-CNN model with 60x45 input images since 14.45 ms can also be a high-speed response that can be used for pupil detection. With higher resolutions such as 120x90, although it may result in a better prediction performance during the training, it may not be as convenient to use in real-time pupil detection applications since the average prediction response time is 40.94.

[0106] Fig. 6H shows the accuracy comparison of quantized and non-quantized models on the LPW test dataset. Both models performed similar RMSE and MAE trends throughout the training. Conventional post-quantization strategies typically reduced float-point model parameters’ precision to integers, decreasing prediction performance. The reason for the similar trends between non-quantized and quantized models was the QAT strategy, in which parameters were quantized during the training.

[0107] As shown in FIG. 6H, the quantized model results in an accurate and robust performance like the non-quantized ResCNN model, even though the quantized model significantly outperformed the non-quantized model in terms of the prediction speed. Specifically, Fig. 6H shows the RMSE and MAE comparison of non-quantized and quantized ResCNN models.

[0108] Indeed, experiments showed that the quantized Res-CNN model trained with the QAT strategy performed well on the test dataset while providing real-time predictions on a resource-constrained single-board computer.

[0109] Discussion

[0110] Orbital Exenteration Prothesis Discussion. The principal drawback of the conventional “gold standard” orbital exenteration prosthesis, even if the cosmetic match is perfect, is the inability to match any synchronous dynamic actions of the contralateral normal eye - eyelid blinking and conjugate eye movements. As a result, the prosthesis is immediately and easily noticeable once the patient blinks or moves the normal eye. The prosthetic eye/eyelid remains stationary or adynamic. This discrepancy is often noticed by observers immediately and is a source of much consternation for the wearer. Psychologically, the wearer still does not feel whole.

[OHl] In Weisson, Ernesto H., et al. “Automated noncontact facial topography mapping, 3-dimensional printing, and silicone casting of orbital prosthesis,” American journal of ophthalmology 220 (2020): 27-36, a workflow to fabricate an exenteration prosthesis using an automated noncontact facial topography mapping 3 -Dimensional printing process was developed. The novel 3-D printed exenteration prosthesis fabrication method has been reduced to practice. FIGS. 7A-7H show the steps of the process including facial topographical mapping, 3D printing molds, injection molding, detailing, making the exenteration prosthesis, finishing details, and the final outcome. Such steps may be implemented into the prothesis production for systems, methods, and devices of this disclosure. This production workflow has the potential to provide an efficient, standardized, reproducible prosthesis and to overcome the principal barriers to an affordable custom device for an underserved population worldwide: access and cost. However, the described silicone orbital prosthesis does not contain integrated movement components. Thus, this disclosure provides systems, methods, devices, and overall improvements on the prosthesis produced by the above-described process. Al Development Discussion. Eye-tracking technologies have gained popularity since they highlight real-time information on essential human sensory systems [1], The gaze of an eye, the size of the pupil, and the frequency of the motion can provide valuable insights into human interactions. The application span of these technologies has dramatically expanded with the recent development of wearable devices, including human-computer interaction [2], biometric recognition [3], and research in consumer psychology [4], Two major technologies have been developed for real-time eye tracking: remote sensor-based eye tracking and head-mounted sensor-based eye tracking [5], While each has advantages and disadvantages, both technologies use a video camera and computer vision to detect and track eye and pupil motion.

[0112] In recent years, CNNs emerged as powerful algorithms that can automatically detect long-range dependencies of a given image [6] and have proven their success as feature extractors in many practical applications [7], The robust and accurate prediction performance of CNNs allowed researchers to develop models with strong generalization ability for pupil detection. Fuhl et al. [8] proposed a dual CNN pipeline for pupil detection. Chinsatit and Saitoh [9] trained a CNN model for pupil detection using infrared eye images captured by a wearable inside-out camera. Vera-Olmos et al. [2] proposed DeepEye, a CNN architecture built with residual connections [10] and atrous convolution [11] for pupil segmentation. Lee et al. [12] proposed a fast ocular recognition method with high accuracy using the ResNet [10] architecture. Yiu et al. [13] proposed DeepVOG, a U-Net [14] based segmentation model for pupil segmentation. Kothari et al. [15] implemented a CNN-based framework for pupil and iris ellipse segmentation. Shi et al. [16] proposed a high-accuracy, pupil tracking CNN model integrated with a Long-Short Term Memory (LSTM) algorithm. Recently, Akinlar et al. [17] proposed a new loss term called ellipse fit error for pupil segmentation using U-Net. Their model trained with standard binary cross entropy loss function with ellipse fit error as the regularization term achieved state-of-the-art results for commonly used datasets such as ExCuSe [18], ElSe [19], and Labelled Pupils in the Wild (LPW) [20], [0113] Although Deep Fully Convolutional Networks (DFCNs) based segmentation frameworks can provide accurate and robust representation for the given pupil center detection problem, they demand high computational resources. As a result, they suffer from slow response time in real-time applications. However, mobile computers with limited computing power may be desirable in various eye tracking applications. Single Board Computers (SBC) such as a Raspberry Pi are among these portable computers, and they usually consist of a mobile central processing unit (CPU) that is an ARM architecture-based processor and lacks a graphical processing unit (GPU). They lack the GPUs mainly used in deep learning-based computer vision applications, and the processing unit of the CPU may be several orders less (e.g., nearly six times slower) than that of a modem computer. One natural solution to avoid these limitations is utilizing CNNs to predict the pupil center location directly rather than creating segmentation maps before pupil detection since a typical segmentation network [14, 21] is designed with an encoder part to extract the features of the given image and a decoder part (typically symmetrical with the encoder) to construct the segmentation maps using those features.

[0114] On the other hand, when using CNNs to predict the pupil center directly, the decoder part is replaced with fully connected layers which utilize features extracted by the encoder to give the final predictions. Such a strategy reduces the number of model parameters and mathematical operations, decreasing the prediction response time. Another method of increasing the prediction speed is adapting a quantization strategy that compresses models by reducing bits per weight [22], Conventional post-quantization strategies such as scalar quantization which reduces the precision of floating-point learned model parameters to integers, can significantly compress the models, thus reducing complexity [23], Although these post-quantization strategies can reduce the complexity, lowering the precision for the model parameters can increase the numerical errors, ultimately leading to a decrease in accuracy [24], One way to address this issue is by quantizing the model parameters throughout the training [22], Quantization Aware Training (QAT) [25] framework can solve the above problems by quantizing all the model parameters in the forward propagation phase while computing gradients using a straight-through estimator (STE) [26, 27], Therefore, this QAT strategy enables the development of robust, accurate, and fast models, particularly for real-time applications.

[0115] The exemplary Al model, in some embodiments, employs an end-to-end, robust CNN model that can generate accurate and real-time pupil center predictions for low computational power applications. The training of the exemplary Al model may leverage the QAT strategy to achieve real-time prediction responses without sacrificing accuracy. Moreover, to increase the prediction performance without increasing the model complexity, the model may include residual connections [10] to avoid the vanishing gradient problem and Squeeze and Excitation networks [28] as a robust mechanism to increase the attention to more valuable features. The model may also integrate Atrous Spatial Pyramid Pooling (ASPP) [29] layer into the transition between convolutional and fully connected layers to capture the long-range dependencies in feature maps while significantly increasing the prediction speed. In the model training, transfer learning may be employed using a synthetic eye image dataset, namely, SynthesEyes Dataset [30], to improve prediction performance further. Experiments show that the developed model with transfer learning and QAT strategies is an improvement to technology that can provide superior performance on prediction time and competitive results in detecting performance compared to state-of-the-art deep learning-based pupil detection algorithms. Therefore, the instant methodology may offer an alternative pupil tracking platform for applications that require additional restrictions such as portability and low computational power.

[0116] The current invention disclosure intends to create a realistic prosthesis that observers cannot discern the person is wearing a prosthesis. In the broadest sense, this involves duplicating the synchronous and conjugate movement of the contralateral healthy eye and the periorbital area, thereby yielding the appearance of fully functioning eye movements and eyelid blinking actions. The edges of the affixed prosthesis to the skin are camouflaged by a custom-designed eyewear frame housing an infra-red camera, battery, and hardware for the integrated system. A well-made orbital exenteration prosthesis with integrated conjugate eye movements and a custom-designed eyewear frame concealing the prosthesis-skin interface to mimic a functional, healthy eye can dramatically improve a patient’s self-esteem and quality of life.

[0117] Unless otherwise expressly stated, it is in no way intended that any method set forth herein be construed as requiring that its steps be performed in a specific order. Accordingly, where a method claim does not actually recite an order to be followed by its steps or it is not otherwise specifically stated in the claims or descriptions that the steps are to be limited to a specific order, it is no way intended that an order be inferred, in any respect. This holds for any possible non-express basis for interpretation, including: matters of logic with respect to arrangement of steps or operational flow; plain meaning derived from grammatical organization or punctuation; the number or type of embodiments described in the specification.

[0118] It will be apparent to those skilled in the art that various modifications and variations can be made without departing from the scope or spirit. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit being indicated by the following claims.

[0119] It is understood that throughout this specification the identifiers “first”, “second”, “third”, “fourth”, “fifth”, “sixth”, and such, are used solely to aid in distinguishing the various components and steps of the disclosed subject matter. The identifiers “first”,

“second”, “third”, “fourth”, “fifth”, “sixth”, and such, are not intended to imply any particular order, sequence, amount, preference, or importance to the components or steps modified by these terms.

[0120] All references cited and discussed in this specification are incorporated herein by reference in their entireties and to the same extent as if each reference was individually incorporated by reference.

REFERENCES

[1] F. Hutmacher, "Why is there so much more research on vision than on any other sensory modality?," Frontiers in psychology, p. 2246, 2019.

[2] F. J. Vera-Olmos, E. Pardo, H. Melero, and N. Malpica, "DeepEye: Deep convolutional network for pupil detection in real environments," Integrated Computer-Aided Engineering, vol. 26, no. 1, pp. 85-95, 2019.

[3] I. A. Saad, L. E. George, and A. A. Tayyar, "Accurate and fast pupil localization using contrast stretching, seed filling and circular geometrical constraints," 2014.

[4] C. W. Jones, W. Sun, H. Boulter, and S. Brown, "3D roughness standard for performance verification of topography instruments for additively-manufactured surface inspection," Measurement Science and Technology, 2022.

[5] J. P. Tao et al., "A Digital Microscreen for the Enhanced Appearance of Ocular Prosthetic Motility (an American Ophthalmological Society Thesis)," American Journal of Ophthalmology, vol. 228, pp. 35-46, 2021.

[6] Y. LeCun et al., "Handwritten digit recognition with a back-propagation network," Advances in neural information processing systems, vol. 2, 1989.

[7] W. Rawat and Z. Wang, "Deep convolutional neural networks for image classification: A comprehensive review," Neural computation, vol. 29, no. 9, pp. 2352-2449, 2017.

[8] W. Fuhl, T. Santini, G. Kasneci, W. Rosenstiel, and E. Kasneci, "Pupilnet v2. 0: Convolutional neural networks for cpu based real time robust pupil detection," arXiv preprint arXiv: 1711.00112, 2017.

[9] W. Chinsatit and T. Saitoh, "CNN-based pupil center detection for wearable gaze estimation system," Applied Computational Intelligence and Soft Computing, vol. 2017, 2017.

[10] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778. [11] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, "Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs," IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 4, pp. 834-848, 2017.

[12] Y. W. Lee, K. W. Kim, T. M. Hoang, M. Arsalan, and K. R. Park, "Deep residual CNNbased ocular recognition based on rough pupil detection in the images by NIR camera sensor," Sensors, vol. 19, no. 4, p. 842, 2019.

[13] Y.-H. Yiu et al., "DeepVOG: Open-source pupil segmentation and gaze estimation in neuroscience using deep learning," Journal of neuroscience methods, vol. 324, p. 108307, 2019.

[14] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in International Conference on Medical image computing and computer-assisted intervention, 2015: Springer, pp. 234-241.

[15] R. S. Kothari, A. K. Chaudhary, R. J. Bailey, J. B. Pelz, and G. J. Diaz, "Ellseg: An ellipse segmentation framework for robust gaze tracking," IEEE Transactions on Visualization and Computer Graphics, vol. 27, no. 5, pp. 2757-2767, 2021.

[16] L. Shi, C. Wang, F. Tian, and H. Jia, "An integrated neural network model for pupil detection and tracking," Soft Computing, vol. 25, no. 15, pp. 10117-10127, 2021.

[17] C. Akinlar, H. K. Kucukkartal, and C. Topal, "Accurate CNN-based pupil segmentation with an ellipse fit error regularization term," Expert Systems with Applications, vol. 188, p. 116004, 2022.

[18] W. Fuhl, T. Kiibler, K. Sippel, W. Rosenstiel, and E. Kasneci, "Excuse: Robust pupil detection in real -world scenarios," in International conference on computer analysis of images and patterns, 2015: Springer, pp. 39-51.

[19] W. Fuhl, T. C. Santini, T. Kiibler, and E. Kasneci, "Else: Ellipse selection for robust pupil detection in real -world environments," in Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications, 2016, pp. 123-130.

[20] M. Tonsen, X. Zhang, Y. Sugano, and A. Bulling, "Labelled pupils in the wild: a dataset for studying pupil detection in unconstrained environments," in Proceedings of the ninth biennial ACM symposium on eye tracking research & applications, 2016, pp. 139-142. [21] V. Badrinarayanan, A. Kendall, and R. Cipolla, "Segnet: A deep convolutional encoderdecoder architecture for image segmentation," IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 12, pp. 2481-2495, 2017.

[22] A. Fan et al., "Training with quantization noise for extreme model compression," arXiv preprint arXiv:2004.07320, 2020.

[23] V. Vanhoucke, A. Senior, and M. Z. Mao, "Improving the speed of neural networks on CPUs," 2011.

[24] P. Stock, A. Joulin, R. Gribonval, B. Graham, and H. Jegou, "And the bit goes down: Revisiting the quantization of neural networks," arXiv preprint arXiv: 1907.05686, 2019.

[25] B. Jacob et al., "Quantization and training of neural networks for efficient integerarithmetic-only inference," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2704-2713.

[26] Y. Bengio, N. Leonard, and A. Courville, "Estimating or propagating gradients through stochastic neurons for conditional computation," arXiv preprint arXiv: 1308.3432, 2013.

[27] M. Courbariaux and Y. Bengio, "Binarynet: Training deep neural networks with weights and activations constrained to+ 1 or- 1. arXiv 2016," arXiv preprint arXiv: 1602.02830.

[28] J. Hu, L. Shen, and G. Sun, "Squeeze-and-excitation networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132-7141.

[29] K. He, X. Zhang, S. Ren, and J. Sun, "Spatial pyramid pooling in deep convolutional networks for visual recognition," IEEE transactions on pattern analysis and machine intelligence, vol. 37, no. 9, pp. 1904-1916, 2015.

[30] E. Wood, T. Baltrusaitis, X. Zhang, Y. Sugano, P. Robinson, and A. Bulling, "Rendering of eyes for eye-shape registration and gaze estimation," in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 3756-3764.

[31] S. Ioffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," in International conference on machine learning, 2015: PMLR, pp. 448-456.

[32] V. Nair and G. E. Hinton, "Rectified linear units improve restricted boltzmann machines," m ' lcml, 2010. [33] M.-H. Guo et aL, "Attention mechanisms in computer vision: A survey," Computational Visual Media, pp. 1-38, 2022.

[34] S. Lazebnik, C. Schmid, and J. Ponce, "Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories," in 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06), 2006, vol. 2: IEEE, pp. 2169-2178.

[35] Y. Wang, B. Liang, M. Ding, and J. Li, "Dense semantic labeling with atrous spatial pyramid pooling and decoder for high -resolution remote sensing imagery," Remote Sensing, vol. 11, no. 1, p. 20, 2018.

[36] J. Wang, P. Lv, H. Wang, and C. Shi, "SAR-U-Net: Squeeze-and-excitation block and atrous spatial pyramid pooling based residual U-Net for automatic liver segmentation in Computed Tomography," Computer Methods and Programs in Biomedicine, vol. 208, p. 106268, 2021.

[37] Y. Sun, L. Wang, Y. Chen, and M. Liu, "Accurate lane detection with atrous convolution and spatial pyramid pooling for autonomous driving," in 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), 2019: IEEE, pp. 642- 647.

[38] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, "Encoder-decoder with atrous separable convolution for semantic image segmentation," in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 801-818.

[39] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv: 1412.6980, 2014.

[40] K. He, X. Zhang, S. Ren, and J. Sun, "Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification," in Proceedings of the IEEE international conference on computer vision, 2015, pp. 1026-1034.

[41] Weisson, Ernesto H., et al., "Automated noncontact facial topography mapping, 3- dimensional printing, and silicone casting of orbital prosthesis," American journal of ophthalmology 220 (2020): 27-36.

Claims

What is claimed is:

1. A system comprising: a prosthetic eye system configured to be placed in an orbital cavity of a person, the prosthetic eye system comprising a motorized eyeball structure configured to move in two or more directions; a sensor system comprising a sensor configured to acquire an image of an eye contralaterally located to the prosthetic eye system; and a controller, the controller comprising: a processor; and a memory having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to: acquire an image of the eye contralaterally located to the prosthetic eye system; determine, via a real-time Al model, a position of a pupil of the eye; and output a command to the prosthetic eye system to move the motorized eyeball structure in a manner corresponding to the eye contralaterally located to the prosthetic eye system.

2. The system of claim 1, wherein the prosthetic eye system comprises: a first motor unit configured to move in a first direction; and a second motor unit operatively coupled to the first motor unit to move in a second direction, wherein the second direction is perpendicular to the first direction.

3. The system of claim 2, wherein the first motor unit and the second motor unit each comprises a DC motor.

4. The system of claim 2, wherein the prosthetic eye system further comprises: a blinking mechanism comprising an eyelid and a third motor operatively coupled to the eyelid to move the eyelid between an open position within the orbital cavity and a closed position covering a front portion of the motorized eyeball structure.

5. The system of claim 4, wherein the processor further outputs a command to the prosthetic eye system to move the eyelid in a manner corresponding to the eye contralaterally located to the prosthetic eye system.

6. The system of claim 1, wherein the sensor system and controller are each located in an eyeglass frame configured to be worn by the person.

7. The system of claim 6, wherein the controller is implemented in a single board computer that is embedded into an ipsilateral temple frame of the eyeglass frame.

8. The system of claim 1, wherein the sensor system comprises an IR camera that is housed at an inferior temporal quadrant of an eyeglass frame.

9. The system of claim 1, wherein the real-time Al model comprises a residual CNN.

10. The system of claim 9, wherein the residual CNN includes residual connections and at least one of Squeeze and Excitation (SE) attention, Atrous Spatial Pyramid Pooling (ASPP), and a combination thereof.

11. The system of claim 9, wherein the residual CNN was trained by quantizing model parameters of the Al model, using a Quantization Aware Training (QAT) operation, to speed up predictions.

12. The system of claim 9, wherein the real-time Al model is configured to execute in real-time time steps of greater than 30 ms per classification on a resource-constrained singleboard computer.

13. A prosthetic eye system configured to be placed in an orbital cavity of a person, the prosthetic eye system comprising: a motorized eyeball structure configured to move in two or more directions comprising: a first motor unit configured to move in a first direction; and a second motor unit operatively coupled to the first motor unit to move in a second direction, wherein the second direction is perpendicular to the first direction.

14. The prosthetic eye system of claim 13, wherein the first motor unit and the second motor unit each comprises a DC motor.

15. An eyeglass frame apparatus configured to be worn by a person, the apparatus comprising: a first ipsilateral temple frame portion; a lens region having an inferior temporal quadrant; a sensor embedded in the inferior temporal quadrant; and a computing device embedded in the ipsilateral temple frame portion and operatively connected to the sensor, the computing device having a controller having a processor; a memory having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to: acquire an image from the sensor; execute a real-time Al model in a real-time loop to determine a position of a pupil of a non-prosthetic eye in the image; and output a command to a prosthetic eye system to move a motorized eyeball structure in a manner corresponding to the non-prosthetic eye.

16. The eyeglass frame apparatus of claim 15 further comprising: a second ipsilateral temple frame portion that is operatively coupled to the lens region; and a rechargeable energy storage unit embedded within the second ipsilateral temple frame portion.

17. A non-transitory computer-readable medium having instructions stored thereon for real-time detecting of a pupil location in an image of an eye, wherein execution for the instructions by a processor, causes the processor to: obtain an image of the eye contralaterally located to a prosthetic eye system; determine, via a real-time Al model executing in a real-time control loop, a position of a pupil of the eye; and output a command to a prosthetic eye system to move a motorized eyeball structure of the prosthetic eye system in a manner corresponding to the eye contralaterally located to the prosthetic eye system.

18. The non-transitory computer-readable medium of claim 17, wherein the real-time Al model comprises a residual CNN.

19. The non-transitory computer-readable medium of claim 18, wherein the residual CNN includes residual connections and at least one of Squeeze and Excitation (SE) attention, Atrous Spatial Pyramid Pooling (ASPP), and a combination thereof.

20. The non-transitory computer-readable medium of claim 18, wherein the residual CNN was trained by quantizing the model parameters to speed up the predictions using Quantization Aware Training (QAT) operation.

21. The non-transitory computer-readable medium of claim 17, wherein the real-time Al model is configured to execute in real-time time steps of greater than 30 ms per classification on a resource-constrained portable computer.

22. A non-transitory computer-readable medium having instructions stored thereon for real-time detecting of an eyelid location in an image of an eye, wherein execution for the instructions by a processor, causes the processor to: obtain an image of the eye contralaterally located to a prosthetic eye system; determine, via a real-time Al model executing in a real-time control loop, a position of an eyelid of the eye; and output a command to the prosthetic eye system to move a motorized eyelid in a manner corresponding to an eyelid of the eye contralaterally located to the prosthetic eye system.