WO2021026948A1 - Optical microscope system and method capable of tracking gaze position in real time - Google Patents

Optical microscope system and method capable of tracking gaze position in real time Download PDF

Info

Publication number
WO2021026948A1
WO2021026948A1 PCT/CN2019/101584 CN2019101584W WO2021026948A1 WO 2021026948 A1 WO2021026948 A1 WO 2021026948A1 CN 2019101584 W CN2019101584 W CN 2019101584W WO 2021026948 A1 WO2021026948 A1 WO 2021026948A1
Authority
WO
WIPO (PCT)
Prior art keywords
gaze
eye tracking
camera
image
tracking camera
Prior art date
Application number
PCT/CN2019/101584
Other languages
French (fr)
Chinese (zh)
Inventor
崔笑宇
丁勇
陈卫兴
魏然
Original Assignee
东北大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 东北大学 filed Critical 东北大学
Publication of WO2021026948A1 publication Critical patent/WO2021026948A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/24—Base structure
    • G02B21/241—Devices for focusing
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/361—Optical details, e.g. image relay to the camera or image sensor
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365—Control or image processing arrangements for digital or video microscopes
    • G02B21/367—Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison

Definitions

  • the invention relates to the technical field of microscope and gaze analysis, and in particular to an optical microscope system and method capable of tracking gaze position in real time.
  • Optical microscopes are currently commonly used instruments for observing microscopic substances. They are mainly composed of objective lenses, eyepieces, stages, and light sources. They have been widely used in biological analysis, material analysis, clinical testing, and pathological diagnosis. According to the optical path structure of the microscope, it can be divided into upright microscope, inverted microscope and stereo microscope.
  • the objective lens of the upright microscope is located above the stage, the sample is placed on the upper surface of the stage, and the upper surface of the sample is observed through the eyepiece;
  • the objective lens of the inverted microscope is located below the stage, and the sample is placed on the lower surface of the stage, passing The eyepiece observes the lower surface of the sample;
  • the stereomicroscope has a larger field of view and depth of field, and the three-dimensional structure of the sample can be observed through the eyepiece.
  • the eyepieces of optical microscopes can be divided into monocular, binocular, trinocular and other types according to the number of lens barrels.
  • the monocular objective lens has only one lens barrel to observe the sample through a single eye;
  • the binocular objective lens has two lens barrels, which can observe the sample through both eyes at the same time;
  • the trinocular objective lens has three lens barrels, two of which are used for binocular observation.
  • a lens tube is placed vertically to place an optical camera to generate digital images or videos of the sample in real time.
  • the main observation method is to observe the field of view area within the eyepiece through the human eye.
  • the human eye is used to observe the characteristics of sample morphology, and the analysis results are given based on experience. Therefore, in addition to the basic operating procedures of the microscope, the observation method of the sample in the field of view, including the observation range, observation sequence, etc., is also extremely important for various applications of optical microscopes.
  • the technical problem to be solved by the present invention is to provide an optical microscope system and method that can track the gaze position in real time in view of the above-mentioned shortcomings of the prior art.
  • the system can analyze the range, sequence, observation time of different positions of the human eye observation sample, etc. Information can also be finally displayed in the form of heat maps.
  • the present invention provides an optical microscope system capable of tracking gaze position in real time, including: a microscope body, a spectroscope, a gaze tracking device, and an image processing module;
  • the microscope body includes a left eyepiece, a left eyepiece observation tube, a right eyepiece, a right eyepiece observation tube, an objective lens, a collimating screw, a stage, a microscope light source, and a microscope camera;
  • the beam splitting device is arranged between the eyepiece and the objective lens, and includes a first beam splitter, a second beam splitter, a first dichroic mirror, a second dichroic mirror and a plane mirror;
  • the gaze tracking device includes a right eye tracking camera, a right infrared light source, a left eye tracking camera, and a left infrared light source;
  • the right eyepiece observation tube, the first dichroic mirror, and the right eye tracking camera are sequentially fixed along the same optical path, the right infrared light source is placed parallel to the right eye tracking camera; the left eyepiece observation tube, the second dichroic mirror, and the left eye tracking camera are along the same optical path Fixed in order, the left infrared light source is placed parallel to the left eye tracking camera; the plane mirror is fixed at the middle of the left eyepiece observation tube and the right eyepiece observation tube and is placed parallel to the second dichroic mirror; the second dichroic mirror and the plane mirror are placed horizontally and with the first dichroic The mirror is placed vertically;
  • the microscope camera, the right eye tracking camera, and the left eye tracking camera are respectively connected to the image processing module;
  • the image processing module includes an image receiving module, a gaze analysis module, and an image generation module; the image receiving module is used to receive image information output by a microscope camera, a right eye tracking camera, and a left eye tracking camera.
  • the output terminal of the image receiving module is connected to The input end of the gaze analysis module is connected; the gaze analysis module is used to receive the image information output by the right eye tracking camera and the left eye tracking camera.
  • the gaze tracking algorithm and the gaze area calibration algorithm it is established by corneal curvature, camera position, and infrared light source position Model the 3D shape of the eyeball, and calculate the line of sight direction of each frame from the corneal reflection images output by the left eye tracking camera and the right eye tracking camera; the output of the 3D shape model of the eyeball is a 3D image with calibrated line of sight.
  • the output terminal of the analysis module is connected with the input terminal of the image generation module; the image generation module is used to receive the 3D image output by the gaze analysis module with the calibrated sight direction, and generate the user's gaze through the user's sight direction in the 3D image
  • the heat map analyzes the way the user observes the image based on the gaze heat map.
  • the cutoff frequency range of the first dichroic mirror and the second dichroic mirror is between 750nm and 900nm.
  • the luminous frequency range of the right infrared light source and the left infrared light source in the gaze tracking device is between 750 nm and 900 nm, and is used to provide illumination for the eye tracking camera.
  • the working band of the right-eye tracking camera and the left-eye tracking camera in the gaze tracking device is the infrared band.
  • the microscope camera in the sight tracking device is located above the first beam splitter.
  • the right eye tracking camera and the right infrared light source in the sight tracking device are located on one side of the first dichroic mirror.
  • the left eye tracking camera and the left infrared light source in the sight tracking device are located on one side of the second dichroic mirror.
  • the present invention provides a method for using an optical microscope that can track the gaze position in real time, which is implemented by the optical microscope system that can track the gaze position in real time, and includes the following steps:
  • Step 1 Place the sample on the stage of the optical microscope
  • Step 2 The objective lens of the optical microscope, the first beam splitter, the second beam splitter, the first dichroic mirror, the second dichroic mirror, the flat mirror, the left eyepiece and the right eyepiece constitute the visual observation optical path; the user observes the sample through the visual observation optical path Observe; the objective lens and the first beam splitter constitute the microscopic image acquisition optical path, the microscope camera uses the microscopic image acquisition optical path to collect the image of the sample; the right eyepiece and the first dichroic mirror constitute the right image acquisition optical path, the left eyepiece and the second dichroic The mirror constitutes the left-side image acquisition light path; the right-eye tracking camera and the left-eye tracking camera use the right-side image acquisition light path and the left-side image acquisition light path to respectively acquire the eye image of the user when observing the sample on the eyepiece;
  • Step 3 The user first adjusts the optical microscope to a low-power lens, adjusts the focus spiral focus, and conducts preliminary observation of the sample, and then switches to a high-power lens, adjusts the focus spiral focus, and observes the sample;
  • Step 4 Based on the real-time images acquired by the microscope camera, right-eye tracking camera, and left-eye tracking camera, establish a three-dimensional shape model of the eyeball through the gaze tracking algorithm and the gaze area calibration algorithm; according to the corneal reflection acquired in real time by the right-eye tracking camera and the left-eye tracking camera Calculate the line of sight direction of the eyes in each frame of the image; the output of the three-dimensional shape model of the eyeball is a three-dimensional image with calibrated line of sight;
  • Step 5 Obtain the user's gaze heat map generated by the user's gaze direction when the user observes the sample image through the calibrated three-dimensional image of the gaze direction, and then merge the gaze heat map with the sample image taken by the microscope camera to analyze the user's observation of the sample the way.
  • the present invention provides an optical microscope system and method capable of tracking the gaze position in real time.
  • the present invention addresses the problem that all current optical microscopes cannot identify the observation area of the human eye.
  • the structure of the trinocular eyepiece Features Combine gaze tracking technology with microscopic imaging technology to design an optical system structure that can capture the eyepiece field of view in real time and binocular vision movement, and propose a real-time calculation method for the core area observed by both eyes. Using the system and method, information such as the range, sequence, and observation time of different positions of the human eye observation sample can be analyzed, and it can also be finally displayed in the form of a heat map.
  • the gaze position information of the eyes can be obtained by extracting and analyzing the captured binocular gaze movement trajectory, which can realize the tracking of the user’s gaze when the user reads the film, and combines the user’s eye movement to identify the position according to its positioning.
  • a heat map of the user’s gaze on the sample is formed, which can also be used for doctors’ teaching and image reading training, speeding up the education and training of pathologists, meeting the needs of a long training period for domestic pathologists, and also using heat maps as Tags, combined with artificial intelligence methods.
  • FIG. 1 is a schematic diagram of a planar structure of an optical microscope system capable of tracking gaze position in real time according to an embodiment of the present invention
  • FIG. 2 is a schematic diagram of a three-dimensional structure of an optical microscope system capable of tracking gaze position in real time according to an embodiment of the present invention
  • FIG. 3 is a schematic diagram of functional modules of an optical microscope system capable of tracking gaze position in real time according to an embodiment of the present invention
  • FIG. 4 is a flowchart of a method of an image processing module provided by an embodiment of the present invention.
  • FIG. 5 is an optical path diagram of an optical microscope system provided by an embodiment of the present invention.
  • Figure 6 is a schematic diagram of a central projection transformation provided by an embodiment of the present invention.
  • the present invention provides an optical microscope system capable of tracking gaze position in real time, including: a microscope body, a spectroscope, a gaze tracking device, and an image processing module;
  • the microscope body includes a left eyepiece 11, a left eyepiece observation tube 18, a right eyepiece 10, a right eyepiece observation tube 17, an objective lens 12, a collimating screw 14, a stage 15, a microscope light source 16, and a microscope camera 13;
  • the beam splitting device is arranged between the eyepiece and the objective lens, and includes a first beam splitter 5, a second beam splitter 6, a first dichroic mirror 7, a second dichroic mirror 8 and a flat mirror 9;
  • the gaze tracking device includes a right eye tracking camera 3, a right infrared light source 1, a left eye tracking camera 4, and a left infrared light source 2;
  • the right infrared light source 1 and the left infrared light source 2 are used to provide infrared light sources for eye tracking cameras;
  • the right eye tracking camera 3 and the left eye tracking camera 4 are used to collect infrared images or videos of the user's eyes;
  • the first beam splitter 5 is used to divide the optical path of the enlarged image of the objective lens 12 into two optical paths, vertical and horizontal;
  • the second beam splitter 6 is used to divide the image optical path from the first beam splitter 5 into two optical paths of 0° and 90° to the right;
  • the first dichroic mirror 7 and the second dichroic mirror 8 are used to transmit infrared light and reflect visible light;
  • the plane mirror 9 is used to reflect the vertical light path emitted by the first beam splitter 5 to the first dichroic mirror 7;
  • the right eyepiece 10 and the left eyepiece 11 are used to observe both eyes of the user, and are also used to collect images or videos of moving eyes of the user;
  • the objective lens 12 is used to amplify the sample
  • the microscope camera 13 is used to collect a sample image or video magnified by the objective lens 8;
  • the right eyepiece observation tube, the first dichroic mirror 7, the right eye tracking camera 3 are sequentially fixed along the same optical path, the right infrared light source 1 and the right eye tracking camera 3 are placed in parallel; the left eyepiece observation tube, the second dichroic mirror 8, the left eye
  • the tracking camera 4 is fixed in sequence along the same optical path, the left infrared light source 2 is placed parallel to the left eye tracking camera 4;
  • the plane mirror 9 is fixed at the middle of the left eyepiece observation tube and the right eyepiece observation tube and placed parallel to the second dichroic mirror 8;
  • the mirror 8 and the plane mirror 9 are placed horizontally and vertically to the first dichroic mirror 7;
  • the microscope camera, the right eye tracking camera, and the left eye tracking camera are respectively connected to the image processing module;
  • the image processing module is embedded in the computer and includes an image receiving module, a gaze analysis module, and an image generating module; the image receiving module is used to receive image information output by a microscope camera, a right eye tracking camera, and a left eye tracking camera, and the image is received
  • the output terminal of the module is connected with the input terminal of the gaze analysis module; the gaze analysis module is used to receive the image information output by the right eye tracking camera and the left eye tracking camera according to the gaze tracking algorithm and the gaze area calibration algorithm through the known parameter corneal curvature , Camera position, infrared light source position to establish a three-dimensional shape modeling of the eyeball, and calculate the line of sight direction of each frame of the eye according to the corneal reflection image output by the left eye tracking camera and the right eye tracking camera; and continuously detect the line of sight It is gaze tracking.
  • the output of the 3D shape model of the eyeball in this part is a 3D image with calibrated gaze direction.
  • the output terminal of the gaze analysis module is connected with the input terminal of the image generation module; the image generation module is used to receive the output of the gaze analysis module
  • the three-dimensional image with the calibrated line of sight direction is used to generate the user’s gaze heat map based on the user’s gaze direction in the three-dimensional image, and analyze the way the user observes the image based on the gaze heat map.
  • the computer includes a desktop computer or a mobile computer composed of core processing chips such as CPU, GPU, ARM, or FPGA; the gaze tracking algorithm and the gaze area calibration algorithm are used to calculate the gaze area according to the direction of the gaze;
  • the cut-off frequency range of the first dichroic mirror and the second dichroic mirror is between 750nm and 900nm, which are used to transmit infrared light and reflect visible light;
  • the luminous frequency range of the right infrared light source and the left infrared light source in the gaze tracking device is between 750 nm and 900 nm, and is used to provide illumination for the eye tracking camera;
  • the working band of the right-eye tracking camera and the left-eye tracking camera in the gaze tracking device is an infrared band
  • the microscope camera in the sight tracking device is located above the first beam splitter;
  • the right eye tracking camera and the right infrared light source in the sight tracking device are located on one side of the first dichroic mirror;
  • the left eye tracking camera and the left infrared light source in the sight tracking device are located on one side of the second dichroic mirror;
  • the present invention provides a method for using an optical microscope that can track the gaze position in real time, implemented by the optical microscope system that can track the gaze position in real time, as shown in FIG. 3, including the following steps:
  • Step 1 Place the sample on the stage of the optical microscope
  • Step 2 The objective lens of the optical microscope, the first beam splitter, the second beam splitter, the first dichroic mirror, the second dichroic mirror, the flat mirror, the left eyepiece and the right eyepiece constitute the visual observation optical path; the user observes the sample through the visual observation optical path Observe; the objective lens and the first beam splitter constitute the microscopic image acquisition optical path, the microscope camera uses the microscopic image acquisition optical path to collect the image of the sample; the right eyepiece and the first dichroic mirror constitute the right image acquisition optical path, the left eyepiece and the second dichroic The mirror constitutes the left-side image acquisition light path; the right-eye tracking camera and the left-eye tracking camera acquire the eye image of the user when the user observes the sample on the eyepiece through the right-side image acquisition light path and the left-side image acquisition light path respectively, as shown in Figure 5;
  • Step 3 The user first adjusts the optical microscope to a low-power lens, adjusts the focus spiral focus, and conducts preliminary observation of the sample, and then switches to a high-power lens, adjusts the focus spiral focus, and observes the sample;
  • Step 4 Based on the real-time images acquired by the microscope camera, right-eye tracking camera, and left-eye tracking camera, establish a three-dimensional shape model of the eyeball through the gaze tracking algorithm and the gaze area calibration algorithm; according to the corneal reflection acquired in real time by the right-eye tracking camera and the left-eye tracking camera Calculate the line of sight direction of the eyes in each frame of the image; and the continuous line of sight detection is the line of sight tracking.
  • the 3D shape model of this part of the eyeball is output as a 3D image with the line of sight direction calibrated, as shown in Figure 4;
  • a reflection point is formed on the cornea of the user's left and right eyes, and the reflection point is the Purkin spot;
  • the left and right eye tracking cameras collect the images of the user's left and right eyes in real time , And extract the vector from the center of the Purkin spot to the center of the pupil in the real-time collected image as the line of sight direction parameter, and then use the eyeball imaging model to estimate the line of sight direction;
  • a mapping model can also be used to estimate the direction of the line of sight, and the gaze direction can be estimated by assuming a spherical eyeball and corneal surface;
  • Step 5 Obtain the user's gaze heat map generated by the user's gaze direction when the user observes the sample image through the calibrated three-dimensional image of the gaze direction, and then merge the gaze heat map with the sample image taken by the microscope camera to analyze the user's observation of the sample Way, as shown in Figure 4.
  • the image of the eyepiece gets the mapping relationship, that is, the mapping relationship between the gaze direction and the visual focus when observing the glass slide, that is, the distance between the position of the human eye and the visual focus of the sample is determined by establishing a three-dimensional coordinate calculation , And then calculate and determine the real-time intersection point of the line of sight of the human eye on the sample.
  • the gaze heat map is the frequency distribution map of the user's visual focus in the image within a defined unit time, which can express the user's behavior in observing and analyzing the microscopic image.
  • the mapping relationship is the central perspective transformation, as shown in Figure 6, E represents the user's eyes, q is the user's line of sight in the image of the eyepiece, and r is the focal point when observing the slide. Establish spatial three-dimensional coordinates for the origin;
  • the coordinates of point r can be expressed as:
  • Pathological diagnosis is the gold standard for cancer diagnosis. In cancer treatment, correct pathological diagnosis is the basis for effective cancer treatment. The results of pathological diagnosis are not only used to judge the nature and classification of the lesion, but also directly determine the surgeon’s surgical method, scope and internal medicine Doctor's medication regimen. Different pathological types, treatment methods, drugs and prognostic effects are quite different. The clear pathological diagnosis gives clinicians the correct guidance, and strives for better prognosis and longer survival time for patients.
  • the training cycle of pathologists is very long. Usually, after reading more than 10,000 pathology cases, they can write a preliminary pathology report independently; after handling 30,000 pathologies, they can review the reports of lower-level doctors; and after handling more than 50,000 cases, can they solve difficult diagnoses. There are currently only more than 20,000 pathologists in my country, but the incidence of malignant tumors has reached more than 10%, and there is a serious shortage of doctors and uneven distribution.
  • the doctor’s gaze heat map of pathological slices generated in this method can be used for doctor’s teaching and reading training, speeding up the education and training of pathologists, meeting the needs of domestic pathologists with a long training cycle, and can also use the heat map as Tags, combined with artificial intelligence methods.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Microscoopes, Condenser (AREA)
  • Eye Examination Apparatus (AREA)

Abstract

An optical microscope system and method capable of tracking the gaze position in real time, which relate to the technical field of microscopes and gaze analysis. A right eyepiece observation tube (17), a first dichroic mirror (7), and a right eye tracking camera (3) are successively fixed along the same optical path, and a right infrared light source (1) is placed parallel to the right eye tracking camera (3); a left eyepiece observation tube (18), a second dichroic mirror (8), and a left eye tracking camera (4) are successively fixed along the same optical path, and a left infrared light source (2) is placed parallel to the left eye tracking camera (4); a plane mirror (9) is fixed to the middle position of the left eyepiece observation tube (18) and the right eyepiece observation tube (17) and is placed parallel to the second dichroic mirror (8); the second dichroic mirror (8) is placed horizontal to the plane mirror (9) and perpendicular to the first dichroic mirror (7); and a microscope camera (13), the right eye tracking camera (3), and the left eye tracking camera (4) are separately connected to an image processing module. The foregoing system may analyze information such as the range, sequence, and observation time of different positions of the human eye observing a sample, and the information may also be finally displayed in the form of a heat map.

Description

一种可实时追踪注视位置的光学显微镜系统及方法Optical microscope system and method capable of tracking gaze position in real time 技术领域Technical field
本发明涉及显微镜和注视分析技术领域，尤其涉及一种可实时追踪注视位置的光学显微镜系统及方法。The invention relates to the technical field of microscope and gaze analysis, and in particular to an optical microscope system and method capable of tracking gaze position in real time.
背景技术Background technique
光学显微镜是目前观测微观物质的常用仪器，主要由物镜、目镜、载物台和光源等几部分组成，在生物分析、材料分析、临床检测、病理诊断等方面已经广泛应用。按照显微镜的光路结构，可以分为正置显微镜、倒置显微镜、体显微镜等类型。其中，正置显微镜的物镜位于载物台上方，样品放置于载物台上表面，通过目镜观察样品的上表面；倒置显微镜的物镜位于载物台下方，样品放置于载物台下表面，通过目镜观察样品下表面；体显微镜具有较大视野和景深，可通过目镜观察样品三维结构。Optical microscopes are currently commonly used instruments for observing microscopic substances. They are mainly composed of objective lenses, eyepieces, stages, and light sources. They have been widely used in biological analysis, material analysis, clinical testing, and pathological diagnosis. According to the optical path structure of the microscope, it can be divided into upright microscope, inverted microscope and stereo microscope. Among them, the objective lens of the upright microscope is located above the stage, the sample is placed on the upper surface of the stage, and the upper surface of the sample is observed through the eyepiece; the objective lens of the inverted microscope is located below the stage, and the sample is placed on the lower surface of the stage, passing The eyepiece observes the lower surface of the sample; the stereomicroscope has a larger field of view and depth of field, and the three-dimensional structure of the sample can be observed through the eyepiece.
光学显微镜的目镜根据镜筒数量主要可分为单目、双目、三目等类型。单目物镜只有一个镜筒，通过单眼观察样品；双目物镜具有两个镜筒，可通过双眼同时观察样品；三目物镜具有三个镜筒，其中左右两个镜筒用于双眼观察，另外一个镜筒垂直放置，用于放置光学相机从而实时生成样品数字图像或视频。The eyepieces of optical microscopes can be divided into monocular, binocular, trinocular and other types according to the number of lens barrels. The monocular objective lens has only one lens barrel to observe the sample through a single eye; the binocular objective lens has two lens barrels, which can observe the sample through both eyes at the same time; the trinocular objective lens has three lens barrels, two of which are used for binocular observation. A lens tube is placed vertically to place an optical camera to generate digital images or videos of the sample in real time.
目前，无论何种类型的光学显微镜，主要观察方式都是通过人眼观察目镜视野范围内的视场区域。此外，在众多应用领域，例如生物分析、材料分析、临床检测、病理分析等，都是利用人眼观察样品形态等特征，根据经验给出分析结果。因此，除显微镜的基本操作流程外，对视场区域内样品的观察方式，包括观察范围、观察顺序等经验，对光学显微镜的各类应用也极为重要。At present, no matter what type of optical microscope, the main observation method is to observe the field of view area within the eyepiece through the human eye. In addition, in many application fields, such as biological analysis, material analysis, clinical testing, pathological analysis, etc., the human eye is used to observe the characteristics of sample morphology, and the analysis results are given based on experience. Therefore, in addition to the basic operating procedures of the microscope, the observation method of the sample in the field of view, including the observation range, observation sequence, etc., is also extremely important for various applications of optical microscopes.
发明概述Summary of the invention
技术问题technical problem
问题的解决方案The solution to the problem
技术解决方案Technical solutions
本发明要解决的技术问题是针对上述现有技术的不足，提供一种可实时追踪注 视位置的光学显微镜系统及方法，本系统可以分析人眼观测样品的范围、顺序、不同位置的观测时间等信息，也能以热图的形式最终展现。The technical problem to be solved by the present invention is to provide an optical microscope system and method that can track the gaze position in real time in view of the above-mentioned shortcomings of the prior art. The system can analyze the range, sequence, observation time of different positions of the human eye observation sample, etc. Information can also be finally displayed in the form of heat maps.
为解决上述技术问题，本发明所采取的技术方案是：In order to solve the above technical problems, the technical solutions adopted by the present invention are:
一方面，本发明提供一种可实时追踪注视位置的光学显微镜系统，包括：显微镜本体、分光装置、视线追踪装置和图像处理模块；In one aspect, the present invention provides an optical microscope system capable of tracking gaze position in real time, including: a microscope body, a spectroscope, a gaze tracking device, and an image processing module;
所述显微镜本体包括左目镜、左目镜观察筒、右目镜、右目镜观察筒、物镜、准焦螺旋、载物台、显微镜光源、显微镜相机；The microscope body includes a left eyepiece, a left eyepiece observation tube, a right eyepiece, a right eyepiece observation tube, an objective lens, a collimating screw, a stage, a microscope light source, and a microscope camera;
所述分光装置设置在目镜和物镜之间，包括第一分光镜、第二分光镜、第一双色镜、第二双色镜和平面镜；The beam splitting device is arranged between the eyepiece and the objective lens, and includes a first beam splitter, a second beam splitter, a first dichroic mirror, a second dichroic mirror and a plane mirror;
所述视线追踪装置包括右眼追踪相机、右红外光源、左眼追踪相机和左红外光源；The gaze tracking device includes a right eye tracking camera, a right infrared light source, a left eye tracking camera, and a left infrared light source;
所述右目镜观察筒、第一双色镜、右眼追踪相机沿同一光路依次固定，右红外光源与右眼追踪相机平行放置；左目镜观察筒、第二双色镜、左眼追踪相机沿同一光路依次固定，左红外光源与左眼追踪相机平行放置；平面镜固定于左目镜观察筒和右目镜观察筒中间位置且与第二双色镜平行放置；第二双色镜与平面镜水平放置且与第一双色镜垂直放置；The right eyepiece observation tube, the first dichroic mirror, and the right eye tracking camera are sequentially fixed along the same optical path, the right infrared light source is placed parallel to the right eye tracking camera; the left eyepiece observation tube, the second dichroic mirror, and the left eye tracking camera are along the same optical path Fixed in order, the left infrared light source is placed parallel to the left eye tracking camera; the plane mirror is fixed at the middle of the left eyepiece observation tube and the right eyepiece observation tube and is placed parallel to the second dichroic mirror; the second dichroic mirror and the plane mirror are placed horizontally and with the first dichroic The mirror is placed vertically;
所述显微镜相机、右眼追踪相机、左眼追踪相机分别与图像处理模块相连接；The microscope camera, the right eye tracking camera, and the left eye tracking camera are respectively connected to the image processing module;
所述图像处理模块包括图像接收模块、注视分析模块、图像生成模块；所述图像接收模块用于接收显微镜相机、右眼追踪相机、左眼追踪相机输出的图像信息，图像接收模块的输出端与注视分析模块的输入端相连接；所述注视分析模块用于接收右眼追踪相机、左眼追踪相机输出的图像信息根据视线追踪算法和注视区域标定算法通过角膜曲率、相机位置、红外光源位置建立眼球三维形状建模，并根据左眼追踪相机和右眼追踪相机输出的角膜反射的图像中计算得出每一帧眼睛的视线方向；眼球三维形状模型输出为标定好视线方向的三维图像，注视分析模块的输出端与图像生成模块的输入端相连接；所述图像生成模块用于接收注视分析模块输出的标定好视线方向的三维图像，通过三维图像中使用者的视线方向生成使用者的凝视热图，根据凝视热图分析使用者观察图像的方式。The image processing module includes an image receiving module, a gaze analysis module, and an image generation module; the image receiving module is used to receive image information output by a microscope camera, a right eye tracking camera, and a left eye tracking camera. The output terminal of the image receiving module is connected to The input end of the gaze analysis module is connected; the gaze analysis module is used to receive the image information output by the right eye tracking camera and the left eye tracking camera. According to the gaze tracking algorithm and the gaze area calibration algorithm, it is established by corneal curvature, camera position, and infrared light source position Model the 3D shape of the eyeball, and calculate the line of sight direction of each frame from the corneal reflection images output by the left eye tracking camera and the right eye tracking camera; the output of the 3D shape model of the eyeball is a 3D image with calibrated line of sight. The output terminal of the analysis module is connected with the input terminal of the image generation module; the image generation module is used to receive the 3D image output by the gaze analysis module with the calibrated sight direction, and generate the user's gaze through the user's sight direction in the 3D image The heat map analyzes the way the user observes the image based on the gaze heat map.
所述的第一双色镜和第二双色镜的截止频率范围在750nm-900nm之间。The cutoff frequency range of the first dichroic mirror and the second dichroic mirror is between 750nm and 900nm.
所述视线追踪装置中的右红外光源和左红外光源发光频率范围在750-900nm之间，用于提供眼追踪相机的照明。The luminous frequency range of the right infrared light source and the left infrared light source in the gaze tracking device is between 750 nm and 900 nm, and is used to provide illumination for the eye tracking camera.
所述视线追踪装置中的右眼追踪相机和左眼追踪相机的工作波段为红外波段。The working band of the right-eye tracking camera and the left-eye tracking camera in the gaze tracking device is the infrared band.
所述视线追踪装置中的显微镜相机位于第一分光镜的上方。The microscope camera in the sight tracking device is located above the first beam splitter.
所述视线追踪装置中的右眼追踪相机和右红外光源位于第一双色镜的一侧。The right eye tracking camera and the right infrared light source in the sight tracking device are located on one side of the first dichroic mirror.
所述视线追踪装置中的左眼追踪相机和左红外光源位于第二双色镜的一侧。The left eye tracking camera and the left infrared light source in the sight tracking device are located on one side of the second dichroic mirror.
另一方面，本发明提供一种可实时追踪注视位置的光学显微镜的使用方法，通过所述的一种可实时追踪注视位置的光学显微镜系统实现，包括如下步骤：On the other hand, the present invention provides a method for using an optical microscope that can track the gaze position in real time, which is implemented by the optical microscope system that can track the gaze position in real time, and includes the following steps:
步骤1：将样品置于光学显微镜的载物台上；Step 1: Place the sample on the stage of the optical microscope;
步骤2：光学显微镜的物镜、第一分光镜、第二分光镜、第一双色镜、第二双色镜、平面镜、左目镜和右目镜构成目视观察光路；使用者通过目视观察光路对样品进行观察；物镜和第一分光镜构成显微图像获取光路，显微镜相机通过显微图像获取光路对样品进行图像采集；右目镜和第一双色镜构成右侧图像获取光路，左目镜和第二双色镜构成左侧图像获取光路；右眼追踪相机和左眼追踪相机通过右侧图像获取光路和左侧图像获取光路分别获取使用者在目镜上观察样品时的眼睛图像；Step 2: The objective lens of the optical microscope, the first beam splitter, the second beam splitter, the first dichroic mirror, the second dichroic mirror, the flat mirror, the left eyepiece and the right eyepiece constitute the visual observation optical path; the user observes the sample through the visual observation optical path Observe; the objective lens and the first beam splitter constitute the microscopic image acquisition optical path, the microscope camera uses the microscopic image acquisition optical path to collect the image of the sample; the right eyepiece and the first dichroic mirror constitute the right image acquisition optical path, the left eyepiece and the second dichroic The mirror constitutes the left-side image acquisition light path; the right-eye tracking camera and the left-eye tracking camera use the right-side image acquisition light path and the left-side image acquisition light path to respectively acquire the eye image of the user when observing the sample on the eyepiece;
步骤3：使用者将光学显微镜先调到低倍镜，调节准焦螺旋聚焦，对样品进行初步观察，再转换到高倍镜，调节准焦螺旋聚焦，对样品进行观察；Step 3: The user first adjusts the optical microscope to a low-power lens, adjusts the focus spiral focus, and conducts preliminary observation of the sample, and then switches to a high-power lens, adjusts the focus spiral focus, and observes the sample;
步骤4：根据显微镜相机、右眼追踪相机、左眼追踪相机实时获取的图像通过视线追踪算法和注视区域标定算法建立眼球三维形状模型；根据右眼追踪相机、左眼追踪相机实时获取的角膜反射的图像中计算得出每一帧眼睛的视线方向；眼球三维形状模型输出为标定好视线方向的三维图像；Step 4: Based on the real-time images acquired by the microscope camera, right-eye tracking camera, and left-eye tracking camera, establish a three-dimensional shape model of the eyeball through the gaze tracking algorithm and the gaze area calibration algorithm; according to the corneal reflection acquired in real time by the right-eye tracking camera and the left-eye tracking camera Calculate the line of sight direction of the eyes in each frame of the image; the output of the three-dimensional shape model of the eyeball is a three-dimensional image with calibrated line of sight;
步骤5：通过标定好视线方向的三维图像得到使用者观察样品图像时视线方向生成的使用者的凝视热图，再将凝视热图与显微镜相机拍摄的样本图像进行融合以分析使用者观察样品的方式。Step 5: Obtain the user's gaze heat map generated by the user's gaze direction when the user observes the sample image through the calibrated three-dimensional image of the gaze direction, and then merge the gaze heat map with the sample image taken by the microscope camera to analyze the user's observation of the sample the way.
发明的有益效果The beneficial effects of the invention
有益效果Beneficial effect
采用上述技术方案所产生的有益效果在于：本发明提供的一种可实时追踪注视位置的光学显微镜系统及方法，本发明针对目前所有光学显微镜无法识别人眼观测区域的问题，根据三目目镜结构特点，将视线追踪技术与显微成像技术结合，设计出可实时捕获目镜视野范围双眼视线移动的光学系统结构，同时提出双眼所观测的核心区域的实时计算方法。利用该系统及方法，可以分析人眼观测样品的范围、顺序、不同位置的观测时间等信息，也可以热图的形式最终展现。可以通过对捕获的双眼视线运动轨迹进行提取和分析，获得双眼的注视位置信息，可以实现在使用者阅片时，对使用者视线的追踪，并结合使用者眼部运动，根据其定位辨识的结果，从而形成使用者对样品的注视热图，也可用于医生的教学和阅片训练，加快病理科医生的教育培训速度，满足国内病理医生培养周期长的需求，同时也可将热图作为标签，与人工智能方法结合使用。The beneficial effects produced by the above technical solution are: the present invention provides an optical microscope system and method capable of tracking the gaze position in real time. The present invention addresses the problem that all current optical microscopes cannot identify the observation area of the human eye. According to the structure of the trinocular eyepiece Features: Combine gaze tracking technology with microscopic imaging technology to design an optical system structure that can capture the eyepiece field of view in real time and binocular vision movement, and propose a real-time calculation method for the core area observed by both eyes. Using the system and method, information such as the range, sequence, and observation time of different positions of the human eye observation sample can be analyzed, and it can also be finally displayed in the form of a heat map. The gaze position information of the eyes can be obtained by extracting and analyzing the captured binocular gaze movement trajectory, which can realize the tracking of the user’s gaze when the user reads the film, and combines the user’s eye movement to identify the position according to its positioning. As a result, a heat map of the user’s gaze on the sample is formed, which can also be used for doctors’ teaching and image reading training, speeding up the education and training of pathologists, meeting the needs of a long training period for domestic pathologists, and also using heat maps as Tags, combined with artificial intelligence methods.
对附图的简要说明Brief description of the drawings
附图说明Description of the drawings
图1为本发明实施例提供的可实时追踪注视位置的光学显微镜系统的平面结构示意图；FIG. 1 is a schematic diagram of a planar structure of an optical microscope system capable of tracking gaze position in real time according to an embodiment of the present invention;
图2为本发明实施例提供的可实时追踪注视位置的光学显微镜系统的立体结构示意图；2 is a schematic diagram of a three-dimensional structure of an optical microscope system capable of tracking gaze position in real time according to an embodiment of the present invention;
图3为本发明实施例提供的可实时追踪注视位置的光学显微镜系统功能模块示意图；3 is a schematic diagram of functional modules of an optical microscope system capable of tracking gaze position in real time according to an embodiment of the present invention;
图4为本发明实施例提供的图像处理模块的方法流程图；4 is a flowchart of a method of an image processing module provided by an embodiment of the present invention;
图5为本发明实施例提供的光学显微镜系统光路图；FIG. 5 is an optical path diagram of an optical microscope system provided by an embodiment of the present invention;
图6为本发明实施例提供的中心投影变换原理图；Figure 6 is a schematic diagram of a central projection transformation provided by an embodiment of the present invention;
图中：1.右红外光源；2.左红外光源；3.右眼追踪相机；4.左眼追踪相机；5.第一分光镜；6.第二分光镜；7.第一双色镜；8.第二双色镜；9.平面镜；10.右目镜；11.左目镜；12.物镜；13.显微镜相机；14.准焦螺旋；15.载物台；16.显微镜光源；17.右目镜观察筒；18.左目镜观察筒。In the figure: 1. Right infrared light source; 2. Left infrared light source; 3. Right eye tracking camera; 4. Left eye tracking camera; 5. First beam splitter; 6. Second beam splitter; 7. First dichroic mirror; 8. Second dichroic mirror; 9. Plane mirror; 10. Right eyepiece; 11. Left eyepiece; 12. Objective lens; 13. Microscope camera; 14. Focus spiral; 15. Stage; 16. Microscope light source; 17. Right Eyepiece observation tube; 18. Left eyepiece observation tube.
发明实施例Invention embodiment
本发明的实施方式Embodiments of the invention
下面结合附图和实施例，对本发明的具体实施方式作进一步详细描述。以下实施例用于说明本发明，但不用来限制本发明的范围。The specific embodiments of the present invention will be described in further detail below in conjunction with the drawings and embodiments. The following examples are used to illustrate the present invention, but not to limit the scope of the present invention.
如图1-图2所示，本实施例的方法如下所述。As shown in Figures 1 to 2, the method of this embodiment is as follows.
一方面，本发明提供一种可实时追踪注视位置的光学显微镜系统，包括：显微镜本体、分光装置、视线追踪装置和图像处理模块；In one aspect, the present invention provides an optical microscope system capable of tracking gaze position in real time, including: a microscope body, a spectroscope, a gaze tracking device, and an image processing module;
所述显微镜本体包括左目镜11、左目镜观察筒18、右目镜10、右目镜观察筒17、物镜12、准焦螺旋14、载物台15、显微镜光源16、显微镜相机13；The microscope body includes a left eyepiece 11, a left eyepiece observation tube 18, a right eyepiece 10, a right eyepiece observation tube 17, an objective lens 12, a collimating screw 14, a stage 15, a microscope light source 16, and a microscope camera 13;
所述分光装置设置在目镜和物镜之间，包括第一分光镜5、第二分光镜6、第一双色镜7、第二双色镜8和平面镜9；The beam splitting device is arranged between the eyepiece and the objective lens, and includes a first beam splitter 5, a second beam splitter 6, a first dichroic mirror 7, a second dichroic mirror 8 and a flat mirror 9;
所述视线追踪装置包括右眼追踪相机3、右红外光源1、左眼追踪相机4和左红外光源2；The gaze tracking device includes a right eye tracking camera 3, a right infrared light source 1, a left eye tracking camera 4, and a left infrared light source 2;
所述右红外光源1和所述左红外光源2，用于为眼追踪相机提供红外光源；The right infrared light source 1 and the left infrared light source 2 are used to provide infrared light sources for eye tracking cameras;
所述右眼追踪相机3和左眼追踪相机4，用于采集用户双眼红外图像或视频；The right eye tracking camera 3 and the left eye tracking camera 4 are used to collect infrared images or videos of the user's eyes;
所述第一分光镜5，用于将物镜12放大图像光路分为垂直和水平两条光路；The first beam splitter 5 is used to divide the optical path of the enlarged image of the objective lens 12 into two optical paths, vertical and horizontal;
所述第二分光镜6，用于将来自所述第一分光镜5的图像光路分为0°和向右90°两条光路；The second beam splitter 6 is used to divide the image optical path from the first beam splitter 5 into two optical paths of 0° and 90° to the right;
所述第一双色镜7和第二双色镜8，用于透射红外光、反射可见光；The first dichroic mirror 7 and the second dichroic mirror 8 are used to transmit infrared light and reflect visible light;
所述平面镜9，用于将第一分光镜5出射的垂直光路反射到第一双色镜7；The plane mirror 9 is used to reflect the vertical light path emitted by the first beam splitter 5 to the first dichroic mirror 7;
所述右目镜10和左目镜11，用于用户双眼观察，同时也用于采集双眼移动图像或视频；The right eyepiece 10 and the left eyepiece 11 are used to observe both eyes of the user, and are also used to collect images or videos of moving eyes of the user;
所述物镜12，用于放大样品；The objective lens 12 is used to amplify the sample;
所述显微镜相机13，用于采集物镜8放大的样品图像或视频；The microscope camera 13 is used to collect a sample image or video magnified by the objective lens 8;
所述右目镜观察筒、第一双色镜7、右眼追踪相机3沿同一光路依次固定，右红外光源1与右眼追踪相机3平行放置；左目镜观察筒、第二双色镜8、左眼追踪相机4沿同一光路依次固定，左红外光源2与左眼追踪相机4平行放置；平面镜9固定于左目镜观察筒和右目镜观察筒中间位置且与第二双色镜8平行放置；第二双色镜8与平面镜9水平放置且与第一双色镜7垂直放置；The right eyepiece observation tube, the first dichroic mirror 7, the right eye tracking camera 3 are sequentially fixed along the same optical path, the right infrared light source 1 and the right eye tracking camera 3 are placed in parallel; the left eyepiece observation tube, the second dichroic mirror 8, the left eye The tracking camera 4 is fixed in sequence along the same optical path, the left infrared light source 2 is placed parallel to the left eye tracking camera 4; the plane mirror 9 is fixed at the middle of the left eyepiece observation tube and the right eyepiece observation tube and placed parallel to the second dichroic mirror 8; The mirror 8 and the plane mirror 9 are placed horizontally and vertically to the first dichroic mirror 7;
所述显微镜相机、右眼追踪相机、左眼追踪相机分别与图像处理模块相连接；The microscope camera, the right eye tracking camera, and the left eye tracking camera are respectively connected to the image processing module;
所述图像处理模块嵌与计算机内，包括图像接收模块、注视分析模块、图像生成模块；所述图像接收模块用于接收显微镜相机、右眼追踪相机、左眼追踪相机输出的图像信息，图像接收模块的输出端与注视分析模块的输入端相连接；所述注视分析模块用于接收右眼追踪相机、左眼追踪相机输出的图像信息根据视线追踪算法和注视区域标定算法通过已知参数角膜曲率、相机位置、红外光源位置建立眼球三维形状建模，并根据左眼追踪相机和右眼追踪相机输出的角膜反射的图像中计算得出每一帧眼睛的视线方向；而连续地不断地视线检测即为视线追踪，此部分的眼球三维形状模型输出为标定好视线方向的三维图像，注视分析模块的输出端与图像生成模块的输入端相连接；所述图像生成模块用于接收注视分析模块输出的标定好视线方向的三维图像，通过三维图像中使用者的视线方向生成使用者的凝视热图，根据凝视热图分析使用者观察图像的方式。所述计算机包括由CPU、GPU、ARM或FPGA等核心处理芯片组成的台式计算机或移动计算机；所述视线追踪算法和注视区域标定算法，用于根据视线方向计算注视区域；The image processing module is embedded in the computer and includes an image receiving module, a gaze analysis module, and an image generating module; the image receiving module is used to receive image information output by a microscope camera, a right eye tracking camera, and a left eye tracking camera, and the image is received The output terminal of the module is connected with the input terminal of the gaze analysis module; the gaze analysis module is used to receive the image information output by the right eye tracking camera and the left eye tracking camera according to the gaze tracking algorithm and the gaze area calibration algorithm through the known parameter corneal curvature , Camera position, infrared light source position to establish a three-dimensional shape modeling of the eyeball, and calculate the line of sight direction of each frame of the eye according to the corneal reflection image output by the left eye tracking camera and the right eye tracking camera; and continuously detect the line of sight It is gaze tracking. The output of the 3D shape model of the eyeball in this part is a 3D image with calibrated gaze direction. The output terminal of the gaze analysis module is connected with the input terminal of the image generation module; the image generation module is used to receive the output of the gaze analysis module The three-dimensional image with the calibrated line of sight direction is used to generate the user’s gaze heat map based on the user’s gaze direction in the three-dimensional image, and analyze the way the user observes the image based on the gaze heat map. The computer includes a desktop computer or a mobile computer composed of core processing chips such as CPU, GPU, ARM, or FPGA; the gaze tracking algorithm and the gaze area calibration algorithm are used to calculate the gaze area according to the direction of the gaze;
所述的第一双色镜和第二双色镜的截止频率范围在750nm-900nm之间，用于透射红外光、反射可见光；The cut-off frequency range of the first dichroic mirror and the second dichroic mirror is between 750nm and 900nm, which are used to transmit infrared light and reflect visible light;
所述视线追踪装置中的右红外光源和左红外光源发光频率范围在750-900nm之间，用于提供眼追踪相机的照明；The luminous frequency range of the right infrared light source and the left infrared light source in the gaze tracking device is between 750 nm and 900 nm, and is used to provide illumination for the eye tracking camera;
所述视线追踪装置中的右眼追踪相机和左眼追踪相机的工作波段为红外波段；The working band of the right-eye tracking camera and the left-eye tracking camera in the gaze tracking device is an infrared band;
所述视线追踪装置中的显微镜相机位于第一分光镜的上方；The microscope camera in the sight tracking device is located above the first beam splitter;
所述视线追踪装置中的右眼追踪相机和右红外光源位于第一双色镜的一侧；The right eye tracking camera and the right infrared light source in the sight tracking device are located on one side of the first dichroic mirror;
所述视线追踪装置中的左眼追踪相机和左红外光源位于第二双色镜的一侧；The left eye tracking camera and the left infrared light source in the sight tracking device are located on one side of the second dichroic mirror;
另一方面，本发明提供一种可实时追踪注视位置的光学显微镜的使用方法，通过所述的一种可实时追踪注视位置的光学显微镜系统实现，如图3所示，包括如下步骤：On the other hand, the present invention provides a method for using an optical microscope that can track the gaze position in real time, implemented by the optical microscope system that can track the gaze position in real time, as shown in FIG. 3, including the following steps:
步骤1：将样品置于光学显微镜的载物台上；Step 1: Place the sample on the stage of the optical microscope;
步骤2：光学显微镜的物镜、第一分光镜、第二分光镜、第一双色镜、第二双色镜、平面镜、左目镜和右目镜构成目视观察光路；使用者通过目视观察光路 对样品进行观察；物镜和第一分光镜构成显微图像获取光路，显微镜相机通过显微图像获取光路对样品进行图像采集；右目镜和第一双色镜构成右侧图像获取光路，左目镜和第二双色镜构成左侧图像获取光路；右眼追踪相机和左眼追踪相机通过右侧图像获取光路和左侧图像获取光路分别获取使用者在目镜上观察样品时的眼睛图像，如图5所示；Step 2: The objective lens of the optical microscope, the first beam splitter, the second beam splitter, the first dichroic mirror, the second dichroic mirror, the flat mirror, the left eyepiece and the right eyepiece constitute the visual observation optical path; the user observes the sample through the visual observation optical path Observe; the objective lens and the first beam splitter constitute the microscopic image acquisition optical path, the microscope camera uses the microscopic image acquisition optical path to collect the image of the sample; the right eyepiece and the first dichroic mirror constitute the right image acquisition optical path, the left eyepiece and the second dichroic The mirror constitutes the left-side image acquisition light path; the right-eye tracking camera and the left-eye tracking camera acquire the eye image of the user when the user observes the sample on the eyepiece through the right-side image acquisition light path and the left-side image acquisition light path respectively, as shown in Figure 5;
步骤3：使用者将光学显微镜先调到低倍镜，调节准焦螺旋聚焦，对样品进行初步观察，再转换到高倍镜，调节准焦螺旋聚焦，对样品进行观察；Step 3: The user first adjusts the optical microscope to a low-power lens, adjusts the focus spiral focus, and conducts preliminary observation of the sample, and then switches to a high-power lens, adjusts the focus spiral focus, and observes the sample;
步骤4：根据显微镜相机、右眼追踪相机、左眼追踪相机实时获取的图像通过视线追踪算法和注视区域标定算法建立眼球三维形状模型；根据右眼追踪相机、左眼追踪相机实时获取的角膜反射的图像中计算得出每一帧眼睛的视线方向；而连续地不断地视线检测即为视线追踪，此部分的眼球三维形状模型输出为标定好视线方向的三维图像，如图4所示；Step 4: Based on the real-time images acquired by the microscope camera, right-eye tracking camera, and left-eye tracking camera, establish a three-dimensional shape model of the eyeball through the gaze tracking algorithm and the gaze area calibration algorithm; according to the corneal reflection acquired in real time by the right-eye tracking camera and the left-eye tracking camera Calculate the line of sight direction of the eyes in each frame of the image; and the continuous line of sight detection is the line of sight tracking. The 3D shape model of this part of the eyeball is output as a 3D image with the line of sight direction calibrated, as shown in Figure 4;
具体步骤为：The specific steps are:
通过设置远离光轴的左、右红外光源在使用者左、右眼的角膜上形成反射点，该反射点即为普尔钦斑；左、右眼追踪相机实时采集使用者左、右眼的图像，并在实时采集图像的内提取普尔钦斑的中心到瞳孔中心的矢量作为视线方向参数，然后利用眼球成像模型估计视线方向；By setting the left and right infrared light sources away from the optical axis, a reflection point is formed on the cornea of the user's left and right eyes, and the reflection point is the Purkin spot; the left and right eye tracking cameras collect the images of the user's left and right eyes in real time , And extract the vector from the center of the Purkin spot to the center of the pupil in the real-time collected image as the line of sight direction parameter, and then use the eyeball imaging model to estimate the line of sight direction;
除却上述方法外还可以使用映射模型来估计视线方向，通过假设球形眼球和角膜表面估计凝视方向；In addition to the above methods, a mapping model can also be used to estimate the direction of the line of sight, and the gaze direction can be estimated by assuming a spherical eyeball and corneal surface;
步骤5：通过标定好视线方向的三维图像得到使用者观察样品图像时视线方向生成的使用者的凝视热图，再将凝视热图与显微镜相机拍摄的样本图像进行融合以分析使用者观察样品的方式，如图4所示。Step 5: Obtain the user's gaze heat map generated by the user's gaze direction when the user observes the sample image through the calibrated three-dimensional image of the gaze direction, and then merge the gaze heat map with the sample image taken by the microscope camera to analyze the user's observation of the sample Way, as shown in Figure 4.
具体方法为：标定使用者在目镜内观察到的视野范围，也就是记录使用者在观察视野边缘时，所述视野范围＝视场数/物镜倍率，视场数、倍率为固定参数；本实施例中视场数为22、物镜倍率为40，则视野范围为0.55mm；视线的方向；在标定完成后，在视野范围内的视线方向都能在左眼追踪相机和右眼追踪相机获取的眼睛的图像(即目镜的影像)得到映射关系，也就是凝视方向与观察载玻片时的视焦点之间的映射关系，即通过建立空间三维坐标计算确定人体眼睛所 在位置相对于样本视焦点的距离，进而计算确定人眼的视线方向在样本上的实时相交点。同样的，在观察载玻片时，不断的得到使用者观察其图像的视焦点，以定义好的时间距离，生成凝视热图。而凝视热图是使用者在定义好的单位时间内在图像中的视焦点频率分布图，可以表达出使用者观察与分析显微图像的行为方式。The specific method is: calibrate the field of view observed by the user in the eyepiece, that is, record the field of view = the number of field of view/the magnification of the objective lens when the user observes the edge of the field of view, the number of field of view and the magnification are fixed parameters; In the example, the field of view is 22 and the objective magnification is 40, the field of view is 0.55mm; the direction of the line of sight; after the calibration is completed, the line of sight in the field of view can be captured by the left eye tracking camera and the right eye tracking camera The image of the eyepiece (ie the image of the eyepiece) gets the mapping relationship, that is, the mapping relationship between the gaze direction and the visual focus when observing the glass slide, that is, the distance between the position of the human eye and the visual focus of the sample is determined by establishing a three-dimensional coordinate calculation , And then calculate and determine the real-time intersection point of the line of sight of the human eye on the sample. Similarly, when observing the slide glass, the focus of the user's observation of the image is continuously obtained, and the gaze heat map is generated with a defined time distance. The gaze heat map is the frequency distribution map of the user's visual focus in the image within a defined unit time, which can express the user's behavior in observing and analyzing the microscopic image.
所述映射关系为中心透视变换，如图6所示，E代表使用者眼睛，q即为在目镜的影像中的使用者的视线方向点，r为观察载玻片时的视焦点，以o为原点建立空间三维坐标；The mapping relationship is the central perspective transformation, as shown in Figure 6, E represents the user's eyes, q is the user's line of sight in the image of the eyepiece, and r is the focal point when observing the slide. Establish spatial three-dimensional coordinates for the origin;
r点的坐标可以表示为：The coordinates of point r can be expressed as:
r＝E+t(q-E)r=E+t(q-E)
其中t为乘法系数；Where t is the multiplication coefficient;
病理诊断是癌症诊断的金标准，在癌症治疗中，正确的病理诊断是癌症有效治疗的基础，病理诊断结果不仅用于判断病变的性质和分类，还直接决定外科医生的手术方法、范围和内科医生的用药方案。不同的病理类型，治疗采用的手段、药物以及预后效果大相径庭。明确的病理诊断给临床医生正确的指导，为患者争取更好的预后和更长的生存时间。Pathological diagnosis is the gold standard for cancer diagnosis. In cancer treatment, correct pathological diagnosis is the basis for effective cancer treatment. The results of pathological diagnosis are not only used to judge the nature and classification of the lesion, but also directly determine the surgeon’s surgical method, scope and internal medicine Doctor's medication regimen. Different pathological types, treatment methods, drugs and prognostic effects are quite different. The clear pathological diagnosis gives clinicians the correct guidance, and strives for better prognosis and longer survival time for patients.
病理医生的培养周期非常长，通常病理阅读1万例以上后，才能独立撰写初步的病理报告；经手3万病理，才能复查下级医生的报告；经手5万例以上，才能解决疑难诊断。我国目前只有2万多名病理科医生，但恶性肿瘤发病率达到10％以上，存在严重的医生短缺和分布不均的状况。本方法中生成的医生对病理切片的注视热图，可用于医生的教学和阅片训练，加快病理科医生的教育培训速度，满足国内病理医生培养周期长的需求，同时也可将热图作为标签，与人工智能方法结合使用。The training cycle of pathologists is very long. Usually, after reading more than 10,000 pathology cases, they can write a preliminary pathology report independently; after handling 30,000 pathologies, they can review the reports of lower-level doctors; and after handling more than 50,000 cases, can they solve difficult diagnoses. There are currently only more than 20,000 pathologists in my country, but the incidence of malignant tumors has reached more than 10%, and there is a serious shortage of doctors and uneven distribution. The doctor’s gaze heat map of pathological slices generated in this method can be used for doctor’s teaching and reading training, speeding up the education and training of pathologists, meeting the needs of domestic pathologists with a long training cycle, and can also use the heat map as Tags, combined with artificial intelligence methods.
最后应说明的是：以上实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明权利要求所限定的范围。Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions recorded in the foregoing embodiments are modified, or some or all of the technical features thereof are equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims (8)

  1. 一种可实时追踪注视位置的光学显微镜系统，其特征在于：包括：显微镜本体、分光装置、视线追踪装置和图像处理模块；An optical microscope system capable of tracking the gaze position in real time, which is characterized by comprising: a microscope body, a spectroscopic device, a gaze tracking device and an image processing module;
    所述显微镜本体包括左目镜、左目镜观察筒、右目镜、右目镜观察筒、物镜、准焦螺旋、载物台、显微镜光源、显微镜相机；The microscope body includes a left eyepiece, a left eyepiece observation tube, a right eyepiece, a right eyepiece observation tube, an objective lens, a collimating screw, a stage, a microscope light source, and a microscope camera;
    所述分光装置设置在目镜和物镜之间，包括第一分光镜、第二分光镜、第一双色镜、第二双色镜和平面镜；The beam splitting device is arranged between the eyepiece and the objective lens, and includes a first beam splitter, a second beam splitter, a first dichroic mirror, a second dichroic mirror and a plane mirror;
    所述视线追踪装置包括右眼追踪相机、右红外光源、左眼追踪相机和左红外光源；The gaze tracking device includes a right eye tracking camera, a right infrared light source, a left eye tracking camera, and a left infrared light source;
    所述右目镜观察筒、第一双色镜、右眼追踪相机沿同一光路依次固定，右红外光源与右眼追踪相机平行放置；左目镜观察筒、第二双色镜、左眼追踪相机沿同一光路依次固定，左红外光源与左眼追踪相机平行放置；平面镜固定于左目镜观察筒和右目镜观察筒中间位置且与第二双色镜平行放置；第二双色镜与平面镜水平放置且与第一双色镜垂直放置；The right eyepiece observation tube, the first dichroic mirror, and the right eye tracking camera are sequentially fixed along the same optical path, the right infrared light source is placed parallel to the right eye tracking camera; the left eyepiece observation tube, the second dichroic mirror, and the left eye tracking camera are along the same optical path Fixed in order, the left infrared light source is placed parallel to the left eye tracking camera; the plane mirror is fixed at the middle of the left eyepiece observation tube and the right eyepiece observation tube and is placed parallel to the second dichroic mirror; the second dichroic mirror and the plane mirror are placed horizontally and with the first dichroic The mirror is placed vertically;
    所述显微镜相机、右眼追踪相机、左眼追踪相机分别与图像处理模块相连接；The microscope camera, the right eye tracking camera, and the left eye tracking camera are respectively connected to the image processing module;
    所述图像处理模块包括图像接收模块、注视分析模块、图像生成模块；所述图像接收模块用于接收显微镜相机、右眼追踪相机、左眼追踪相机输出的图像信息，图像接收模块的输出端与注视分析模块的输入端相连接；所述注视分析模块用于接收右眼追踪相机、左眼追踪相机输出的图像信息根据视线追踪算法和注视区域标定算法通过角膜曲率、相机位置、红外光源位置建立眼球三维形状建模，并根据左眼追踪相机和右眼追踪相机输出的角膜反射的图像中计算得出每一帧眼睛的视线方向；眼球三维形状模型输出为标定好视线方向的三维图像，注视分析模块的输出端与图像生成模块的输入端相连接；所述图像生成模块用于接收注视分析模块输出的标定好视线方向的三维图像，通过三维图像中使用者 的视线方向生成使用者的凝视热图，根据凝视热图分析使用者观察图像的方式。The image processing module includes an image receiving module, a gaze analysis module, and an image generation module; the image receiving module is used to receive image information output by a microscope camera, a right eye tracking camera, and a left eye tracking camera. The output terminal of the image receiving module is connected to The input end of the gaze analysis module is connected; the gaze analysis module is used to receive the image information output by the right eye tracking camera and the left eye tracking camera. According to the gaze tracking algorithm and the gaze area calibration algorithm, it is established by corneal curvature, camera position, and infrared light source position Model the 3D shape of the eyeball, and calculate the line of sight direction of each frame from the corneal reflection images output by the left eye tracking camera and the right eye tracking camera; the output of the 3D shape model of the eyeball is a 3D image with calibrated line of sight. The output terminal of the analysis module is connected with the input terminal of the image generation module; the image generation module is used to receive the 3D image output by the gaze analysis module with the calibrated sight direction, and generate the user's gaze through the user's sight direction in the 3D image The heat map analyzes the way the user observes the image based on the gaze heat map.
  2. 根据权利要求1所述的一种可实时追踪注视位置的光学显微镜系统，其特征在于：所述的第一双色镜和第二双色镜的截止频率范围在750nm-900nm之间。The optical microscope system capable of tracking the gaze position in real time according to claim 1, wherein the cut-off frequency range of the first dichroic mirror and the second dichroic mirror is between 750nm and 900nm.
  3. 根据权利要求1所述的一种可实时追踪注视位置的光学显微镜系统，其特征在于：所述视线追踪装置中的右红外光源和左红外光源发光频率范围在750-900nm之间，用于提供眼追踪相机的照明。The optical microscope system capable of tracking the gaze position in real time according to claim 1, wherein the right infrared light source and the left infrared light source in the gaze tracking device have a luminous frequency range between 750-900nm for providing Illumination of the eye tracking camera.
  4. 根据权利要求1所述的一种可实时追踪注视位置的光学显微镜系统，其特征在于：所述视线追踪装置中的右眼追踪相机和左眼追踪相机的工作波段为红外波段。The optical microscope system capable of tracking the gaze position in real time according to claim 1, wherein the working band of the right-eye tracking camera and the left-eye tracking camera in the gaze tracking device is an infrared band.
  5. 根据权利要求1所述的一种可实时追踪注视位置的光学显微镜系统，其特征在于：所述视线追踪装置中的显微镜相机位于第一分光镜的上方。The optical microscope system capable of tracking the gaze position in real time according to claim 1, wherein the microscope camera in the gaze tracking device is located above the first beam splitter.
  6. 根据权利要求1所述的一种可实时追踪注视位置的光学显微镜系统，其特征在于：所述视线追踪装置中的右眼追踪相机和右红外光源位于第一双色镜的一侧。The optical microscope system capable of tracking the gaze position in real time according to claim 1, wherein the right eye tracking camera and the right infrared light source in the gaze tracking device are located on one side of the first dichroic mirror.
  7. 根据权利要求1所述的一种可实时追踪注视位置的光学显微镜系统，其特征在于：所述视线追踪装置中的左眼追踪相机和左红外光源位于第二双色镜的一侧。The optical microscope system capable of tracking gaze position in real time according to claim 1, wherein the left-eye tracking camera and the left infrared light source in the gaze tracking device are located on one side of the second dichroic mirror.
  8. 一种可实时追踪注视位置的光学显微镜的使用方法，通过权利要求1所述的一种可实时追踪注视位置的光学显微镜系统实现，包括如下步骤：A method for using an optical microscope capable of tracking the gaze position in real time is implemented by the optical microscope system capable of tracking the gaze position in real time according to claim 1, comprising the following steps:
    步骤1：将样品置于光学显微镜的载物台上；Step 1: Place the sample on the stage of the optical microscope;
    步骤2：光学显微镜的物镜、第一分光镜、第二分光镜、第一双色镜、第二双色镜、平面镜、左目镜和右目镜构成目视观察光路；使用者通过目视观察光路对样品进行观察；物镜和第一分光镜构成显微图像获取光路，显微镜相机通过显微图像获取光路对样品 进行图像采集；右目镜和第一双色镜构成右侧图像获取光路，左目镜和第二双色镜构成左侧图像获取光路；右眼追踪相机和左眼追踪相机通过右侧图像获取光路和左侧图像获取光路分别获取使用者在目镜上观察样品时的眼睛图像；Step 2: The objective lens of the optical microscope, the first beam splitter, the second beam splitter, the first dichroic mirror, the second dichroic mirror, the flat mirror, the left eyepiece and the right eyepiece constitute the visual observation optical path; the user observes the sample through the visual observation optical path Observe; the objective lens and the first beam splitter constitute the microscopic image acquisition optical path, the microscope camera uses the microscopic image acquisition optical path to collect the image of the sample; the right eyepiece and the first dichroic mirror constitute the right image acquisition optical path, the left eyepiece and the second dichroic The mirror constitutes the left-side image acquisition optical path; the right-eye tracking camera and the left-eye tracking camera obtain the eye image of the user when the user observes the sample on the eyepiece through the right-side image acquisition optical path and the left-side image acquisition optical path;
    步骤3：使用者将光学显微镜先调到低倍镜，调节准焦螺旋聚焦，对样品进行初步观察，再转换到高倍镜，调节准焦螺旋聚焦，对样品进行观察；Step 3: The user first adjusts the optical microscope to a low-power lens, adjusts the focus spiral focus, and conducts preliminary observation of the sample, and then switches to a high-power lens, adjusts the focus spiral focus, and observes the sample;
    步骤4：根据显微镜相机、右眼追踪相机、左眼追踪相机实时获取的图像通过视线追踪算法和注视区域标定算法建立眼球三维形状模型；根据右眼追踪相机、左眼追踪相机实时获取的角膜反射的图像中计算得出每一帧眼睛的视线方向；眼球三维形状模型输出为标定好视线方向的三维图像；Step 4: Based on the real-time images acquired by the microscope camera, right-eye tracking camera, and left-eye tracking camera, establish a three-dimensional shape model of the eyeball through the gaze tracking algorithm and the gaze area calibration algorithm; according to the corneal reflection acquired in real time by the right-eye tracking camera and the left-eye tracking camera Calculate the line of sight direction of the eyes in each frame of the image; the output of the three-dimensional shape model of the eyeball is a three-dimensional image with calibrated line of sight;
    步骤5：通过标定好视线方向的三维图像得到使用者观察样品图像时视线方向生成的使用者的凝视热图，再将凝视热图与显微镜相机拍摄的样本图像进行融合以分析使用者观察样品的方式。Step 5: Obtain the user's gaze heat map generated by the user's gaze direction when the user observes the sample image through the calibrated three-dimensional image of the gaze direction, and then merge the gaze heat map with the sample image taken by the microscope camera to analyze the user's observation of the sample the way.
PCT/CN2019/101584 2019-08-14 2019-08-20 Optical microscope system and method capable of tracking gaze position in real time WO2021026948A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910747909.6 2019-08-14
CN201910747909.6A CN110441901A (en) 2019-08-14 2019-08-14 It is a kind of can real-time tracing watch the optical microscope system and method for position attentively

Publications (1)

Publication Number Publication Date
WO2021026948A1 true WO2021026948A1 (en) 2021-02-18

Family

ID=68435284

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/101584 WO2021026948A1 (en) 2019-08-14 2019-08-20 Optical microscope system and method capable of tracking gaze position in real time

Country Status (2)

Country Link
CN (1) CN110441901A (en)
WO (1) WO2021026948A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230104656A1 (en) * 2020-02-21 2023-04-06 Seetrue Technologies Oy Gaze tracking

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111103679A (en) * 2019-12-03 2020-05-05 苏州福尼亚医疗科技有限公司 External double-path synchronous parallel light 3D image real-time acquisition device and system of microscope
CN111616800B (en) * 2020-06-09 2023-06-09 电子科技大学 Ophthalmic surgery navigation system
CA3188627A1 (en) * 2020-07-27 2022-02-03 Agilent Technologies, Inc. Annotation data collection using gaze-based tracking
CN114903635A (en) * 2021-02-10 2022-08-16 苏州速迈医学科技股份有限公司 Dental microscopic diagnosis and treatment system
CN112926521B (en) * 2021-03-30 2023-01-24 青岛小鸟看看科技有限公司 Eyeball tracking method and system based on light source on-off

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103458770A (en) * 2011-03-18 2013-12-18 Smi创新传感技术有限公司 Optical measuring device and method for capturing at least one parameter of at least one eye wherein an illumination characteristic is adjustable
CN106659541A (en) * 2014-03-19 2017-05-10 直观外科手术操作公司 Medical Devices, Systems, And Methods Integrating Eye Gaze Tracking For Stereo Viewer
CN107145086A (en) * 2017-05-17 2017-09-08 上海青研科技有限公司 A kind of Eye-controlling focus device and method for exempting from calibration
US20180024341A1 (en) * 2015-02-09 2018-01-25 Arizona Board Of Regents On Behalf Of The University Of Arizona Augmented stereoscopic microscopy
WO2018042413A1 (en) * 2016-08-28 2018-03-08 Siegel Gabriel A system for histological examination of tissue specimens
CN108270996A (en) * 2016-12-30 2018-07-10 安讯士有限公司 Watch thermal map attentively
CN109564778A (en) * 2016-08-01 2019-04-02 威里利生命科学有限责任公司 Pathology data capture

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE59504515D1 (en) * 1994-10-26 1999-01-21 Leica Mikroskopie Sys Ag MICROSCOPE, ESPECIALLY SURGICAL MICROSCOPE
GB9823977D0 (en) * 1998-11-02 1998-12-30 Scient Generics Ltd Eye tracking method and apparatus
US6607527B1 (en) * 2000-10-17 2003-08-19 Luis Antonio Ruiz Method and apparatus for precision laser surgery
EP3267892A4 (en) * 2015-03-13 2019-01-23 Richard Awdeh Methods and systems for regitration using a microscope insert
TWI569040B (en) * 2015-05-07 2017-02-01 尚立光電股份有限公司 Autofocus head mounted display device
CN106168853B (en) * 2016-06-23 2019-10-01 中国科学技术大学 A kind of free space wear-type gaze tracking system

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103458770A (en) * 2011-03-18 2013-12-18 Smi创新传感技术有限公司 Optical measuring device and method for capturing at least one parameter of at least one eye wherein an illumination characteristic is adjustable
CN106659541A (en) * 2014-03-19 2017-05-10 直观外科手术操作公司 Medical Devices, Systems, And Methods Integrating Eye Gaze Tracking For Stereo Viewer
US20180024341A1 (en) * 2015-02-09 2018-01-25 Arizona Board Of Regents On Behalf Of The University Of Arizona Augmented stereoscopic microscopy
CN109564778A (en) * 2016-08-01 2019-04-02 威里利生命科学有限责任公司 Pathology data capture
WO2018042413A1 (en) * 2016-08-28 2018-03-08 Siegel Gabriel A system for histological examination of tissue specimens
CN108270996A (en) * 2016-12-30 2018-07-10 安讯士有限公司 Watch thermal map attentively
CN107145086A (en) * 2017-05-17 2017-09-08 上海青研科技有限公司 A kind of Eye-controlling focus device and method for exempting from calibration

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230104656A1 (en) * 2020-02-21 2023-04-06 Seetrue Technologies Oy Gaze tracking
US12086309B2 (en) * 2020-02-21 2024-09-10 Seetrue Technologies Oy Gaze tracking

Also Published As

Publication number Publication date
CN110441901A (en) 2019-11-12

Similar Documents

Publication Publication Date Title
WO2021026948A1 (en) Optical microscope system and method capable of tracking gaze position in real time
CN106236006B (en) 3D optical molecular image laparoscope imaging systems
US11609416B2 (en) Imaging systems with micro optical element arrays and methods of specimen imaging
WO2021000466A1 (en) Optical coherence tomography augmented reality-based surgical microscope imaging system and method
CN107111122A (en) Amplification and associated device in ophthalmologic operation, system and method
JP7154044B2 (en) slit lamp microscope and ophthalmic system
US11986240B2 (en) Surgical applications with integrated visualization camera and optical coherence tomography
CN112512402B (en) Slit-lamp microscope and ophthalmic system
JP7228342B2 (en) slit lamp microscope and ophthalmic system
CN103745257B (en) A kind of computational methods of cone cell density based on image recognition
WO2021256130A1 (en) Slit lamp microscope
US20230333389A1 (en) Head wearable virtual image module for superimposing virtual image on real-time image
Palmer et al. Glare-free retinal imaging using a portable light field fundus camera
US20230359007A1 (en) Imaging systems with micro optical element arrays and methods of specimen imaging
JP7133950B2 (en) Ophthalmic system, ophthalmic information processing device, program, and recording medium
CN108294726A (en) Binocular fundus camera imaging optical system
CN207168484U (en) Binocular fundus camera imaging optical system
JP7237219B2 (en) Ophthalmic system, ophthalmic information processing device, program, and recording medium
CN108577801A (en) Portable stereoscopic is imaged fundus camera optical system
Coppin Mathematical modeling of a light field fundus camera and its applications to retinal imaging
김진 Development of an AI-Enhanced AR Navigation System for Precise Mastoidectomy
CN112731647A (en) Device for realizing three-dimensional imaging by using conventional endoscope
Kong et al. Indirect ophthalmoscopic stereo video system using three-dimensional LCD

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19941269

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19941269

Country of ref document: EP

Kind code of ref document: A1