CN118542635A - Endoscope control system based on eyeball movement recognition - Google Patents
Endoscope control system based on eyeball movement recognition Download PDFInfo
- Publication number
- CN118542635A CN118542635A CN202410995207.0A CN202410995207A CN118542635A CN 118542635 A CN118542635 A CN 118542635A CN 202410995207 A CN202410995207 A CN 202410995207A CN 118542635 A CN118542635 A CN 118542635A
- Authority
- CN
- China
- Prior art keywords
- endoscope
- eyeball
- movement
- mode
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 210000005252 bulbus oculi Anatomy 0.000 title claims abstract description 85
- 230000033001 locomotion Effects 0.000 title claims abstract description 80
- 230000004424 eye movement Effects 0.000 claims abstract description 33
- 238000012545 processing Methods 0.000 claims abstract description 26
- 238000012544 monitoring process Methods 0.000 claims abstract description 12
- 238000000034 method Methods 0.000 claims abstract description 9
- 230000004044 response Effects 0.000 claims abstract description 7
- 210000001508 eye Anatomy 0.000 claims description 25
- 230000005540 biological transmission Effects 0.000 claims description 10
- 230000008859 change Effects 0.000 claims description 9
- 238000005286 illumination Methods 0.000 claims description 9
- 238000013527 convolutional neural network Methods 0.000 claims description 8
- 230000007246 mechanism Effects 0.000 claims description 8
- 230000006870 function Effects 0.000 claims description 6
- 230000000007 visual effect Effects 0.000 claims description 6
- 238000005516 engineering process Methods 0.000 claims description 5
- 238000004891 communication Methods 0.000 claims description 4
- 238000013135 deep learning Methods 0.000 claims description 4
- 238000006073 displacement reaction Methods 0.000 claims description 4
- 230000001965 increasing effect Effects 0.000 claims description 4
- 230000001133 acceleration Effects 0.000 claims description 3
- 238000012937 correction Methods 0.000 claims description 3
- 230000002708 enhancing effect Effects 0.000 claims description 3
- 230000005236 sound signal Effects 0.000 claims description 3
- 238000005096 rolling process Methods 0.000 abstract 1
- 230000009471 action Effects 0.000 description 5
- 238000001514 detection method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000001105 regulatory effect Effects 0.000 description 3
- 230000003213 activating effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 210000000744 eyelid Anatomy 0.000 description 2
- 238000001356 surgical procedure Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000000149 argon plasma sintering Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000004397 blinking Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 230000001276 controlling effect Effects 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000002405 diagnostic procedure Methods 0.000 description 1
- 238000003331 infrared imaging Methods 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000012978 minimally invasive surgical procedure Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 210000001747 pupil Anatomy 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Landscapes
- Endoscopes (AREA)
Abstract
The invention discloses an endoscope control system based on eyeball movement recognition, wherein a multidimensional eyeball movement recognition unit is arranged in a head-mounted display device, and the head-mounted display device is provided with an infrared camera and is used for capturing multidimensional movement data of eyeballs of a user in real time, and the multidimensional eyeball movement data comprises up-and-down movement, left-and-right movement and forward-and-backward rolling; the endoscope space positioning and control unit is electrically connected with the multi-dimensional eyeball motion recognition unit and receives eyeball motion data from the recognition unit; the direct eye movement control of the endoscope function is realized by integrating the double infrared camera system, the image processing and the LED infrared light source with the multipoint layout in the head-mounted display device, and the position, the angle and the focal length of the endoscope are adjusted by continuously monitoring and analyzing the eyeball movement of an operator, so that the control method adapts to the instant requirement in the operation, and not only improves the accuracy and the response speed of the operation, but also obviously improves the convenience of the operation and the overall efficiency of the operation.
Description
Technical Field
The invention relates to the technical field of endoscope control, in particular to an endoscope control system based on eyeball movement identification.
Background
In modern medical practice, endoscopes are widely used in a variety of minimally invasive surgical and diagnostic procedures. The manual operation of the endoscope requires a doctor to adjust the position and focal length of the endoscope while performing the operation, and the control method is complex in operation, and is easy to cause fatigue of an operator in the long-time operation process, and due to the fatigue and limitation of the manual operation, the stability and accuracy of the endoscope are difficult to maintain by using the conventional control method.
Based on this, there is a need for an endoscope control system based on eye movement recognition.
Disclosure of Invention
The invention aims to provide an endoscope control system based on eyeball movement identification, which is used for realizing direct eye movement control of an endoscope function by integrating a double-infrared camera system, image processing and a multi-point layout LED infrared light source in a head-mounted display device, and adjusting the position, angle and focal length of an endoscope by continuously monitoring and analyzing the eyeball movement of an operator so as to adapt to the instant requirement in an operation.
In order to achieve the above object, the present invention provides the following solutions:
An endoscope control system based on eyeball motion recognition comprises a multidimensional eyeball motion recognition unit, an endoscope space positioning and control unit and an operation mode intelligent selection unit;
The multidimensional eyeball movement identification unit, the endoscope space positioning and control unit and the operation mode intelligent selection unit are interconnected through an internal data processing network to realize the instant transmission and processing of data and commands;
The multi-dimensional eyeball movement recognition unit is arranged in the head-mounted display device, and the head-mounted display device is provided with an infrared camera and is used for capturing multi-dimensional movement data of eyeballs of a user in real time, wherein the multi-dimensional movement data comprise up-down movement, left-right movement and front-back scrolling;
The infrared camera and the image processing module work in a combined mode and are used for identifying a specific eyeball movement direction from the captured eyeball image;
The endoscope space positioning and controlling unit is electrically connected with the multidimensional eyeball motion recognition unit, receives eyeball motion data from the recognition unit, and adjusts the space position and focal length of the endoscope according to the received data so as to match the sight and the operation intention of a user;
the control unit controls the multi-dimensional position adjustment and focal length change of the endoscope;
The intelligent operation mode selection unit is provided with a mode identification module which is used for identifying the operation intention sent by a user through a specific eyeball movement mode and switching the observation mode, the treatment mode and the working mode of the illumination adjustment mode of the endoscope according to the identification result;
In this embodiment, the multidimensional eye movement recognition unit further includes an LED infrared light source for enhancing the accuracy of eye movement capturing, the LED infrared light source is configured at the inner side of the head-mounted display device and is aligned to the eye region of the user, the LED infrared light source adopts a multi-point layout manner, and surrounds the eye region of the user, so as to uniformly irradiate the eyeball and eliminate the influence caused by the change of ambient light, so as to improve the stability and accuracy of eye tracking, the wavelength of the LED infrared light source is set to 850 nm, and the wavelength is safe to the eye and has minimal interference under most ambient light conditions;
The method comprises the steps of providing two infrared cameras, wherein the high-precision infrared cameras have a frame rate of 1000 FPS and a resolution of at least 1920x1080 pixels, and are arranged at a specific angle to realize cross capture of eyeball movement, and the capture precision of the movement is enhanced through stereoscopic vision, so that each tiny movement of the eyeball can be accurately recorded;
The captured eye movement image is analyzed and parsed in real time by an image processing module that uses a Convolutional Neural Network (CNN) for deep learning to accurately identify and track eye movement from a complex background.
In this embodiment, the endoscope spatial positioning and control unit includes a linear driver for realizing linear displacement of the endoscope, the driving precision of which reaches ±0.01 mm, and a rotation mechanism for adjusting the rotation angle of the endoscope according to the received eyeball movement data, the rotation precision being ±0.1 degrees;
Wherein the linear driver and the rotating mechanism adopt a PID (proportion-integral-derivative) control algorithm for fine adjustment response, and the parameters of the PID controller are dynamically adjusted according to the speed and the acceleration of eyeball movement so as to adapt to rapid changes in operation, in particular: by calculation formula WhereinIs the coefficient of the rotational response and,Is the compensation coefficient of the compensation coefficient,Is the rotation angle data of the eyeball,The change rate of the eyeball rotation angle is the change rate, so that the ultra-precise control of the endoscope is realized;
Further comprising automatic error correction, automatically adjusting parameters of the PID controller by continuously monitoring the difference between the actual and the expected position of the endoscope, to minimize operational errors, ensure surgical accuracy, and in particular to achieve accurate control of the endoscope by:
continuously monitoring the difference between the actual position and the expected position of the endoscope to obtain a position error value;
Judging whether the position error value exceeds a first preset error threshold (+ -0.02 mm) to obtain a first judgment result;
if the first judgment result is yes, increasing the adjustment amplitude of the PID controller so as to quickly reduce the position error;
if the first judgment result is negative, further judging whether the position error value is smaller than or equal to a second preset error threshold (+ -0.01 mm);
If yes, maintaining the current parameter setting of the PID controller, and maintaining a fine tuning state to gradually optimize the position accuracy;
If not, reducing the adjustment amplitude of the PID controller;
in this embodiment, the operation mode intelligent selection unit is provided with an acoustic feedback module, and when it is recognized that the user switches operation modes through eye movements, an audio signal is sent out to remind the operator that the modes have been switched.
In this embodiment, the head-mounted display device displays the current view image of the endoscope in real time; the display unit employs an OLED screen with a resolution of at least 2K, providing a refresh rate of at least 120Hz to reduce visual delay and blur.
In this embodiment, the system further includes an ambient light sensing unit for detecting the light intensity of the operating environment and automatically adjusting the brightness of the head-mounted display device and the LED infrared light source, so as to ensure that the accuracy of eye tracking can be maintained under any illumination condition.
In this embodiment, the image processing module includes a function of performing color enhancement processing on the captured eyeball image to improve the recognition rate of the eyeball characteristics, especially in a low-light environment.
In this embodiment, the system is configured with a wireless data transmission unit for implementing data communication between the module units; the wireless transmission unit supports at least Wi-Fi and bluetooth technologies.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
the invention provides an endoscope control system based on eyeball movement identification, which is characterized in that a double-infrared camera system, image processing and LED infrared light sources with multi-point layout are integrated in a head-mounted display device to realize direct eye movement control of an endoscope function, and the position, angle and focal length of the endoscope are adjusted by continuously monitoring and analyzing the eyeball movement of an operator to adapt to the instant requirement in operation.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flow chart of a control method of adaptive infrared imaging detection according to embodiment 1 of the present invention;
FIG. 2 is a schematic diagram showing the light intensity and automatic adjustment determination of the environment according to embodiment 1 of the present invention;
fig. 3 is a schematic diagram of intelligent selection and judgment of an operation mode according to embodiment 1 of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention aims to provide an endoscope control system based on eyeball movement identification, which is used for realizing direct eye movement control of an endoscope function by integrating a double-infrared camera system, image processing and a multi-point layout LED infrared light source in a head-mounted display device, and adjusting the position, angle and focal length of an endoscope by continuously monitoring and analyzing the eyeball movement of an operator so as to adapt to the instant requirement in an operation.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
Examples
As shown in fig. 1, an endoscope control system based on eye movement recognition in the present embodiment includes a multidimensional eye movement recognition unit, an endoscope space positioning and control unit, and an operation mode intelligent selection unit;
The multidimensional eyeball motion recognition unit, the endoscope space positioning and control unit and the operation mode intelligent selection unit are interconnected through an internal data processing network to realize the instant transmission and processing of data and commands, the network adopts an Ethernet-based real-time communication protocol (EtherCAT or PROFINET) which is specially designed for high-speed and low-delay data exchange, the protocol supports high-speed data transmission, the delay is as low as a few microseconds, and the instant reaction and synchronization among the units of the system are ensured;
The multi-dimensional eyeball movement recognition unit is arranged in the head-mounted display device, and the head-mounted display device is provided with an infrared camera and is used for capturing multi-dimensional movement data of eyeballs of a user in real time, wherein the multi-dimensional movement data comprise up-down movement, left-right movement and front-back scrolling; specifically, the vertical movement of eyeballs is captured through double infrared cameras, each infrared camera is provided with image processing, the movement speed and angle of the eyeballs can be calculated in real time, data are converted into control signals, the tilting motor of the endoscope is accurately regulated, a micro stepping technology is used for motor control, extremely fine tilting angle adjustment is allowed, the tilting speed is dynamically regulated according to the movement speed of the eyeballs, smoothness and accuracy of visual field adjustment are guaranteed, the movement of the eyeballs from side to side is captured in real time by the high-precision infrared cameras, specific movement parameters are obtained through image processing software analysis, the parameters are transmitted to a rotation control unit through a data link, the rotation control unit controls one or more high-precision rotation servo motors, the orientation of the endoscope is regulated according to specific instructions of the movement of the eyeballs, the response time of the rotation commands is not more than millisecond level, the real-time of operation is guaranteed, the specific movement pattern of the endoscope is identified through rapid pupil fluctuation detection of the eyeballs, the movement of the endoscope is analyzed to advance and retreat control commands of the endoscope, the control system adjusts the endoscope by using an accurate linear driver, the adjustment of the endoscope is based on the high-precision track system, stable movement path setting is guaranteed, and physical interference in surgery is reduced;
The multi-dimensional eye movement recognition unit further includes an LED infrared light source for enhancing eye movement capturing accuracy, the LED infrared light source being disposed at an inner side of the head-mounted display device to be directed to an eye region of a user, the LED infrared light source being disposed in a multi-point layout manner to surround the eye region of the user, to uniformly illuminate the eye and eliminate influence due to variation of ambient light, to enhance stability and accuracy of eye tracking, the wavelength of the LED infrared light source being set to 850 nm, the LED infrared light source having a wavelength of 850 nm being selected based on consideration of safety to eyes and minimization of interference of ambient light, light having the wavelength being hardly visible to human eyes, thus not causing discomfort of vision or interference with normal vision of the user, meanwhile, under various ambient light conditions, the interference of 850 and nm wavelength infrared light on the existing illumination is extremely small, the stability of a system is enhanced, the LED infrared light sources on the inner side of the head-mounted display device are distributed in multiple points and encircle the whole eye area, the distribution mode can uniformly irradiate the eyeballs of a user from all angles, the sufficient infrared light irradiation can be obtained no matter how the eyes of the user move, dead angles and blind areas are reduced, each LED infrared light source is provided with a micro lens, light rays can be precisely oriented to specific parts of the eyeballs, the orientation is beneficial to improving the service efficiency of the light sources, light scattering and loss are reduced, and the light rays are ensured to directly act on key areas required by eyeball tracking;
as shown in fig. 2, the system further comprises an ambient light sensing unit for detecting the light intensity of the operation environment and automatically adjusting the brightness of the head-mounted display device and the LED infrared light source, so as to ensure that the accuracy of eye tracking can be maintained under any illumination condition; ambient light sensing unit use WhereinIs the intensity of the ambient light,Is an adjustment coefficient;
automatic adjustment logic for automatically adjusting brightness of the head-mounted display device and the LED infrared light source by the ambient light sensing unit:
Initial judgment: judging whether the current ambient light intensity is lower than a first preset intensity (200 lux);
If the intensity is lower than the first preset intensity: the backlight brightness of a display screen in the head-mounted display device and the intensity of an LED infrared light source are increased, so that the accuracy of eyeball tracking and the visibility of the display screen in a dark environment are ensured;
if the intensity is not lower than the first preset intensity: then enter the second stage to judge;
and (3) second-stage judgment: continuing to judge whether the ambient light intensity is lower than a second preset intensity (500 lux);
If yes: keeping the current brightness settings of the head-mounted display device and the LED infrared light source unchanged, because the current brightness is sufficient to provide a good eye tracking and visual experience;
if no: the brightness of the LED infrared light source is reduced, so that discomfort of eyes or influence on the performance of the eyeball tracking sensor caused by the generation of excessively strong infrared light in a brighter environment is avoided;
real-time data from the light sensor is continuously received, and the brightness of the LED infrared light source and the brightness of the display screen are adjusted, so that the eye tracking effect and the visual comfort of a user are optimized.
The method comprises the steps of providing two infrared cameras, wherein the high-precision infrared cameras have a frame rate of 1000 FPS and a resolution of at least 1920x1080 pixels, and are arranged at a specific angle to realize cross capture of eyeball movement, and the capture precision of the movement is enhanced through stereoscopic vision, so that each tiny movement of the eyeball can be accurately recorded;
The captured eye movement image is analyzed and parsed in real time by an image processing module that uses a Convolutional Neural Network (CNN) for deep learning to accurately identify and track eye movement from a complex background.
The infrared camera and the image processing module work in a combined mode and are used for identifying a specific eyeball movement direction from the captured eyeball image;
As shown in fig. 3, the operation mode intelligent selection unit is configured with a mode recognition module for recognizing an operation intention of a user through a specific eye movement mode and switching an observation mode, a treatment mode and a working mode of an illumination adjustment mode of the endoscope according to a recognition result;
the mode recognition module integrates advanced image processing technology and a deep learning framework, can recognize and distinguish different eye movement modes through training a large amount of eyeball movement data, quickly blinks, continuously gazes and quickly moves eyeballs, each action is associated with a specific operation mode, and the combination of a Convolutional Neural Network (CNN) and a cyclic neural network (RNN) is adopted to improve the processing effect on time series data, so that the quick continuous eye movement modes and the intention thereof can be recognized more accurately;
Fast blink is defined as closing the eyelid twice or more in a short time (1-2 seconds) and in the present system is configured as a switch trigger signal for activating or switching to a treatment mode of a specific surgical mode, capturing the fast closing and opening actions of the eyelid by the infrared camera, the frequency and number of these actions being analyzed by the image processing algorithm, and upon identifying a fast blink mode meeting preset conditions, initiating a mode switching protocol by the internal logic controller.
Continuous fixation refers to fixation of an eyeball in one direction or object for a long time, in a surgical mode, continuous fixation is used for activating an observation mode of an enlarged or specific view, and by analyzing the position stability of the eyeball and the duration of fixation in real time, the observation intention of a user can be judged, and for continuous fixation, the system can adjust the focal length or illumination intensity of an endoscope so as to optimize the view quality.
The fast movement of the eye, used in surgery to rapidly switch the viewing area, the action is used in the system to rapidly adjust the direction of the endoscope to follow the user's gaze movement, which tracks the dynamic path and speed of the eye in real time through the infrared camera, and when a rapid and obvious eye jump action is detected, instantly adjusts the position and angle of the endoscope to match the user's gaze direction.
Judging conditions of intelligent switching:
Initial detection stage
Detecting an eyeball movement pattern: continuously monitoring the dynamic state of the eyeball at a high frame rate through an infrared camera, and judging whether the eyeball meets the preset operation mode switching condition or not;
Judging whether the eyeball movement is continuous twice rapid blinking or not to obtain a first judgment result;
If the first judgment result is yes, entering the next judgment stage;
if the first judgment result is negative, continuing monitoring.
Deep judging stage
Further confirming the handover requirement: based on the preliminary determination, the system further evaluates whether a mode switch is required.
Judging whether the current operation mode is a target mode or not;
if the target mode is already the target mode, the switching is not performed;
if not, the system is ready to switch modes.
Requesting the user to confirm: to prevent mishandling, the system requests the user to confirm the mode switch by displaying or voice prompts within the head mounted display device.
Sending a sound prompt to a user, inquiring whether to confirm to switch to the treatment mode;
if the user confirms through a specific eye movement (rapid blink upwards), performing mode switching;
if the user cancels by a specific eye movement (rapid blink down), no switching is performed, and returns to the normal monitoring state.
Performing mode switching
Once the user confirms the switch, the system immediately adjusts the endoscope settings, switching to the corresponding surgical mode;
According to the user confirmation, the system adjusts the focal length, illumination and position of the endoscope, ensures compliance with the requirements of the treatment mode, and displays the current mode as "treatment mode".
In this embodiment, the endoscope spatial positioning and control unit includes a linear driver for realizing linear displacement of the endoscope, the driving precision of which reaches ±0.01 mm, and a rotation mechanism for adjusting the rotation angle of the endoscope according to the received eyeball movement data, the rotation precision being ±0.1 degrees;
Wherein the linear driver and the rotating mechanism adopt a PID (proportion-integral-derivative) control algorithm for fine adjustment response, and the parameters of the PID controller are dynamically adjusted according to the speed and the acceleration of eyeball movement so as to adapt to rapid changes in operation, in particular: by calculation formula WhereinIs the coefficient of the rotational response and,Is the compensation coefficient of the compensation coefficient,Is the rotation angle data of the eyeball,The change rate of the eyeball rotation angle is the change rate, so that the ultra-precise control of the endoscope is realized;
Further comprising automatic error correction, automatically adjusting parameters of the PID controller by continuously monitoring the difference between the actual and the expected position of the endoscope, to minimize operational errors, ensure surgical accuracy, and in particular to achieve accurate control of the endoscope by:
continuously monitoring the difference between the actual position and the expected position of the endoscope to obtain a position error value;
Judging whether the position error value exceeds a first preset error threshold (+ -0.02 mm) to obtain a first judgment result;
if the first judgment result is yes, increasing the adjustment amplitude of the PID controller so as to quickly reduce the position error;
if the first judgment result is negative, further judging whether the position error value is smaller than or equal to a second preset error threshold (+ -0.01 mm);
If yes, maintaining the current parameter setting of the PID controller, and maintaining a fine tuning state to gradually optimize the position accuracy;
If not, reducing the adjustment amplitude of the PID controller;
in this embodiment, the operation mode intelligent selection unit is provided with an acoustic feedback module, and when it is recognized that the user switches operation modes through eye movements, an audio signal is sent out to remind the operator that the modes have been switched.
In this embodiment, the head-mounted display device displays the current view image of the endoscope in real time; the display unit employs an OLED screen with a resolution of at least 2K, providing a refresh rate of at least 120Hz to reduce visual delay and blur.
In this embodiment, the image processing module includes a function of performing color enhancement processing on the captured eyeball image to improve the recognition rate of the eyeball characteristics, especially in a low-light environment.
In this embodiment, the system is configured with a wireless data transmission unit for implementing data communication between the module units; the wireless transmission unit supports at least Wi-Fi and bluetooth technologies.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.
Claims (8)
1. An endoscope control system based on eyeball movement identification is characterized in that: comprising
The device comprises a multidimensional eyeball motion recognition unit, an endoscope space positioning and control unit and an operation mode intelligent selection unit;
The multidimensional eyeball motion recognition unit, the endoscope space positioning and control unit and the operation mode intelligent selection unit are interconnected through an internal data processing network to realize the instant transmission and processing of data and commands;
The multi-dimensional eyeball motion recognition unit is arranged in the head-mounted display device, and the head-mounted display device is provided with an infrared camera and is used for capturing multi-dimensional motion data of eyeballs of a user in real time, wherein the multi-dimensional motion data comprises up-down motion, left-right motion and front-back scrolling;
the infrared camera and the image processing module work in a combined mode and are used for identifying a specific eyeball movement direction from the captured eyeball image;
The endoscope space positioning and controlling unit is electrically connected with the multi-dimensional eyeball motion recognition unit, receives eyeball motion data from the recognition unit, and adjusts the space position and focal length of the endoscope according to the received data so as to match the sight and the operation intention of a user;
The control unit controls the multi-dimensional position adjustment and focal length change of the endoscope;
The intelligent operation mode selection unit is provided with a mode identification module which is used for identifying the operation intention sent by a user through a specific eyeball movement mode and switching the observation mode, the treatment mode and the working mode of the illumination adjustment mode of the endoscope according to the identification result.
2. An endoscope control system based on eye movement recognition according to claim 1 and wherein: the multi-dimensional eye movement recognition unit further comprises an LED infrared light source for enhancing eye movement capturing accuracy, wherein the LED infrared light source is arranged on the inner side of the head-mounted display device and is aligned to the eye area of a user, the LED infrared light source surrounds the eye area of the user in a multi-point layout mode so as to uniformly irradiate the eyeballs and eliminate the influence caused by the change of ambient light, the stability and accuracy of eye tracking are improved, the wavelength of the LED infrared light source is set to 850 nm, and the wavelength is safe to the eyes and has the least interference under most ambient light conditions;
The method comprises the steps of providing two infrared cameras, wherein the high-precision infrared cameras have a frame rate of 1000 FPS and a resolution of at least 1920x1080 pixels, and are arranged at a specific angle to realize cross capture of eyeball movement, and the capture precision of the movement is enhanced through stereoscopic vision, so that each tiny movement of an eyeball can be accurately recorded;
The captured eye movement image is analyzed in real time by the image processing module, which uses Convolutional Neural Network (CNN) for deep learning, and accurately identifies and tracks eye movement from complex background.
3. An endoscope control system based on eye movement recognition according to claim 1 and wherein: the endoscope space positioning and controlling unit comprises a linear driver, a rotating mechanism and a rotating mechanism, wherein the linear driver is used for realizing linear displacement of the endoscope, the driving precision of the linear driver reaches +/-0.01 millimeter, and the rotating mechanism is used for adjusting the rotating angle of the endoscope according to received eyeball motion data, and the rotating precision is +/-0.1 degree;
Wherein the linear driver and the rotating mechanism adopt a proportional-integral-derivative (PID) -based control algorithm for fine adjustment response, and the parameters of the PID controller are dynamically adjusted according to the speed and the acceleration of eyeball movement so as to adapt to rapid change in operation;
Further comprising automatic error correction, automatically adjusting parameters of the PID controller by continuously monitoring the difference between the actual and the expected position of the endoscope, to minimize operational errors, ensure surgical accuracy, and in particular to achieve accurate control of the endoscope by:
a position sensor is arranged on the endoscope or a supporting structure thereof, the movement distance and the angle of the endoscope are measured, the position feedback is provided for a control unit by converting mechanical displacement into an electric signal, the control unit receives the electric signal, and the difference between the actual position and the expected (reference) position of the endoscope is continuously monitored to obtain a position error value;
judging whether the position error value exceeds a first preset error threshold (+ -0.02 mm) or not, and obtaining a first judgment result;
If the first judgment result is yes, increasing the adjustment amplitude of the PID controller so as to quickly reduce the position error;
if the first judgment result is negative, further judging whether the position error value is smaller than or equal to a second preset error threshold (+ -0.01 mm);
If yes, maintaining the current parameter setting of the PID controller, and maintaining a fine tuning state to gradually optimize the position accuracy;
If not, reducing the adjustment amplitude of the PID controller.
4. An endoscope control system based on eye movement recognition according to claim 1 and wherein: the intelligent surgical mode selection unit is provided with an acoustic feedback module, and sends out an audio signal when the user is identified to switch the surgical mode through eyeball movement so as to remind the operator that the mode is switched.
5. An endoscope control system based on eye movement recognition according to claim 1 and wherein: the head-mounted display device displays the current view image of the endoscope in real time; the display unit employs an OLED screen with a resolution of at least 2K, providing a refresh rate of at least 120Hz to reduce visual delay and blur.
6. An endoscope control system based on eye movement recognition according to claim 1 and wherein: the system further comprises an ambient light sensing unit for detecting the light intensity of the operating environment and automatically adjusting the brightness of the head-mounted display device and the LED infrared light source, so as to ensure that the accuracy of eye tracking can be maintained under any illumination condition.
7. An endoscope control system based on eye movement recognition according to claim 1 and wherein: the image processing module comprises a function of performing color enhancement processing on the captured eyeball image so as to improve the identification rate of eyeball characteristics, especially in a low-light environment.
8. An endoscope control system based on eye movement recognition according to claim 1 and wherein: the system is configured with a wireless data transmission unit for realizing data communication between the module units; the wireless transmission unit supports at least Wi-Fi and bluetooth technologies.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410995207.0A CN118542635B (en) | 2024-07-24 | Endoscope control system based on eyeball movement recognition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410995207.0A CN118542635B (en) | 2024-07-24 | Endoscope control system based on eyeball movement recognition |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118542635A true CN118542635A (en) | 2024-08-27 |
| CN118542635B CN118542635B (en) | 2024-10-22 |
Family
ID=
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106456148A (en) * | 2014-03-19 | 2017-02-22 | 直观外科手术操作公司 | Medical devices, systems, and methods using eye gaze tracking |
| CN108065904A (en) * | 2018-01-02 | 2018-05-25 | 京东方科技集团股份有限公司 | Endoscopic system and its control method |
| CN108472095A (en) * | 2015-12-29 | 2018-08-31 | 皇家飞利浦有限公司 | The system of virtual reality device, controller and method are used for robotic surgical |
| CN114041103A (en) * | 2019-05-29 | 2022-02-11 | 直观外科手术操作公司 | Operating mode control system and method for computer-assisted surgery system |
| US20240074827A1 (en) * | 2022-09-05 | 2024-03-07 | Seoul National University Hospital | Headset-based vision device for transmitting multiple medical images and providing surgical assist function |
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106456148A (en) * | 2014-03-19 | 2017-02-22 | 直观外科手术操作公司 | Medical devices, systems, and methods using eye gaze tracking |
| CN108472095A (en) * | 2015-12-29 | 2018-08-31 | 皇家飞利浦有限公司 | The system of virtual reality device, controller and method are used for robotic surgical |
| CN108065904A (en) * | 2018-01-02 | 2018-05-25 | 京东方科技集团股份有限公司 | Endoscopic system and its control method |
| CN114041103A (en) * | 2019-05-29 | 2022-02-11 | 直观外科手术操作公司 | Operating mode control system and method for computer-assisted surgery system |
| US20240074827A1 (en) * | 2022-09-05 | 2024-03-07 | Seoul National University Hospital | Headset-based vision device for transmitting multiple medical images and providing surgical assist function |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110502100B (en) | Virtual reality interaction method and device based on eye movement tracking | |
| US10033994B2 (en) | Systems and methods for differentiating between dominant and weak eyes in 3D display technology | |
| US11844573B2 (en) | Active visual alignment stimuli in fundus photography | |
| RU2661027C2 (en) | Method and system for treatment of visual impairment | |
| US20190235624A1 (en) | Systems and methods for predictive visual rendering | |
| US9486386B2 (en) | Using a 3D display to train a weak eye | |
| US20210325649A1 (en) | System and method to automatically adjust illumination during a microsurgical procedure | |
| JPH02164335A (en) | Eyeball motion analyzing device | |
| WO2016072272A1 (en) | Eyesight examination device, and head mounted-type display device | |
| CN105431765A (en) | Focal length adjustment | |
| CN110568627A (en) | Control method of zoom glasses | |
| CN118542635B (en) | Endoscope control system based on eyeball movement recognition | |
| JP2004337348A (en) | Eye movement analysis system | |
| EP3919963B1 (en) | Gaze tracking apparatus and systems | |
| CN118542635A (en) | Endoscope control system based on eyeball movement recognition | |
| JP2021129993A (en) | Ophthalmologic apparatus | |
| JP2021137130A (en) | Medical image processing device and medical observation system | |
| EP4083682B1 (en) | Head-mountable display apparatus and methods | |
| CN106856560A (en) | Image capturing device for operation and image capturing method thereof | |
| JP2011050479A (en) | Visual function measuring device and visual function training device | |
| CN115875652A (en) | VR equipment and adjusting method thereof, doctor operating table and surgical robot system | |
| US20230309819A1 (en) | Ophthalmologic apparatus | |
| US20230065296A1 (en) | Eye-tracking using embedded electrodes in a wearable device | |
| GB2613084A (en) | Gaze tracking apparatus and systems | |
| JP2002010980A (en) | Ophthalmological system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |