CN117530654A - Real-time binocular pupil inspection system and detection method - Google Patents

Abstract

The invention discloses a real-time binocular pupil inspection system which comprises an imaging module, an image acquisition module, a pupil position detection module, a pupil extraction module, a binocular distance measurement module and a pupil measurement module. The imaging module comprises a light source and two groups of mutually isolated binocular infrared cameras, the two groups of binocular infrared cameras are used for shooting the left eye and the right eye of a tester in real time, and the binocular infrared camera corresponding to each eye can acquire two images at each moment; the image acquisition module is used for enhancing the contrast of the image; the pupil position detection module adopts a trained lightweight target detection model Yolo to detect pupil positions of the left eye and the right eye of the tester in real time; the pupil extraction module adopts a trained deep learning semantic segmentation model FastSCNN to extract an exit pupil area near the detected pupil position; the pupil measuring and calculating module is used for measuring and calculating the pupil diameter, the pupil shape, tracking the pupil sight and detecting blink. The invention can simultaneously detect the eyes and has reliable results.

Description

Real-time binocular pupil inspection system and detection method

Technical Field

The invention relates to the field of pupil detection, in particular to a real-time binocular pupil detection system and a real-time binocular pupil detection method.

Background

Pupil examination is a conventional medical examination procedure that prescreens related ocular and neurological diseases by observing and measuring pupil size, shape and response. Traditional pupil examination requires a physician to manually measure pupil size and observe pupil responses, which is subjective and error-prone and requires a long examination time. To solve these problems, pupillometers have been developed.

The pupil meter is medical equipment based on a computer vision technology, and can automatically detect and measure the pupil size and response by running a program related to pupil examination, so that the examination time is greatly shortened, and the accuracy and reliability of the examination are improved. Pupillometer can also record pupil change data and diagnose and monitor the progression of ocular and neurological disorders by comparing the data. The pupillometer has an increasingly wide application range, can be used for ophthalmic examination in hospitals and clinics, can be applied to the fields of emergency treatment, military and the like, and helps doctors and emergency personnel to rapidly diagnose eye and nervous system diseases.

However, the existing pupillometers are all examined on an eye-by-eye basis, which results in the inability to observe one eye while the other eye is also being examined for pupillary responses. In addition, the bench pupillometer requires fixing the chin of the patient, and the doctor needs to frequently adjust the height of the chin support for different patients, thereby reducing examination efficiency. In addition, the patient may blink and move eyes due to light stimulation during the examination, thereby resulting in inaccurate examination results.

Disclosure of Invention

Aiming at the defects of the prior art, the invention provides a real-time binocular pupil inspection system and a real-time binocular pupil inspection method, which can simultaneously complete the binocular pupil appearance inspection in real time, automatically identify the positions of eyeballs and pupils, rapidly and accurately measure the pupil size and response, and perform eye tracking detection, eliminate subjectivity and error of the traditional manual measurement, and greatly shorten the inspection time.

The aim of the invention is achieved by the following technical scheme:

the real-time binocular pupil inspection system comprises an imaging module, an image acquisition module, a pupil position detection module, a pupil extraction module, a binocular distance measurement module and a pupil measurement module;

the imaging module comprises a light source and two groups of mutually isolated binocular infrared cameras, wherein the light source is used for providing a light source to stimulate a through hole of a tester during testing; the two groups of mutually isolated binocular infrared cameras are used for shooting the left eye and the right eye of a tester in real time, the binocular infrared camera corresponding to each eye can acquire two images at each moment, and the acquired images are input into an infrared image sequence in pairs;

the image acquisition module receives infrared image sequences acquired by two binocular infrared cameras, and preprocessing the images by using histogram equalization to enhance the contrast of the images;

the pupil position detection module inputs the images which are processed by the image acquisition module into a trained lightweight target detection model Yolo, and detects pupil positions of the left eye and the right eye of a tester in real time;

the pupil extraction module adopts a trained deep learning semantic segmentation model FastSCNN to extract an exit pupil area near the detected pupil position;

the binocular distance measuring module is used for calculating the actual distance between each pupil and the corresponding binocular infrared camera;

the pupil measuring and calculating module comprises a pupil diameter calculating sub-module, a pupil shape judging sub-module and a pupil sight tracking sub-module; the pupil diameter calculation submodule is used for calculating the actual diameter of the pupil; the pupil shape judging submodule is used for carrying out ellipse fitting on the extracted pupil area and determining parameters of the ellipse; the pupil sight tracking sub-module is used for calculating the actual movement distance of the pupil in the test process and performing blink detection.

Further, the pupil position includes pixel coordinates of the pupil center, a pixel representation of the pupil region width and height.

Further, the calculation formula for calculating the actual distance from each pupil to the corresponding binocular infrared camera is as follows:

d＝f·b/s

wherein f is the focal length of the binocular infrared camera, b is the optical center distance between two lenses of the binocular infrared camera, s is the parallax distance of the pixel coordinates of the centers of pupils of the left eye and the right eye, and the unit is a pixel.

Further, the specific process of calculating the actual diameter size of the pupil by the pupil diameter calculating submodule includes:

obtaining the actual diameter of the pupil through matrix operation size=d.M.P according to the width w (the unit is a pixel), wherein P is the pixel difference of the left boundary point and the right boundary point on the same horizontal line of the pupil area in the horizontal direction of the pixel coordinates; m is an internal reference matrix of the camera.

Further, when the pupil shape judging submodule performs ellipse fitting on the extracted pupil area, an ellipse fitting method based on a least square method is adopted, namely, an ellipse parameter is determined by minimizing the distance from a point to an ellipse.

Further, the pupil sight tracking sub-module calculates the actual movement distance of the pupil through a phase correlation algorithm.

Further, the device also comprises a light source control module for controlling the brightness and duration of the light source.

Further, the light source is an LED.

Further, the system also comprises a data statistics module, which is used for recording the real-time pupil size, shape and pupil position information into a database.

A real-time binocular pupil inspection method, the method being implemented based on a binocular pupil inspection system, the method comprising the steps of:

step one: the method comprises the steps of starting a light source to stimulate eyes of a tester, shooting left eyes and right eyes of the tester in real time by adopting two independent binocular infrared cameras respectively, enabling each binocular infrared camera to simultaneously acquire two images of corresponding eyes at each moment, and sending two images into an image acquisition module in a group of two images;

step two: preprocessing an image by using histogram equalization by the image acquisition module, and enhancing the contrast of the image;

step three: inputting the images processed by the image acquisition module into a trained lightweight target detection model Yolo by the pupil position detection module, detecting pupil positions of the left eye and the right eye of a tester in real time, performing blink detection, and if the current frame is judged to be in a blink state, skipping other steps, and executing the first step; when the fact that the current pupil position is not located in the edge area of the infrared image is detected, executing the fourth step;

step four: extracting an exit pupil region near the detected pupil position by the pupil extraction module by adopting a trained deep learning semantic segmentation model FastSCNN;

step five: calculating the actual distance from each pupil to the corresponding binocular infrared camera by the binocular distance measuring module, and executing the step six when the actual distance is in a preset distance range;

step six: and measuring the actual diameter of the pupil by the pupil measuring and calculating module, performing ellipse fitting on the extracted pupil area, and calculating the actual movement distance of the pupil in the test process.

The beneficial effects of the invention are as follows:

1. the real-time binocular pupil inspection system and the real-time binocular pupil inspection method can simultaneously perform binocular measurement, and solve the problem that the change of the pupil size of the other side is not recorded while one eye is stimulated in the existing method when the pupil is inspected to react to light.

2. According to the real-time binocular pupil inspection system and the real-time binocular pupil inspection method, when the pupils are inspected to react to light, the offset (unit: mm) of the current pupils relative to the movement at the beginning of inspection is recorded in real time, whether the pupils of an observed person are in the monitoring range of the camera or not and whether the distance from the pupils to the camera is in a reasonable range or not can be monitored in real time, and re-measurement is needed when the distance exceeds the range, so that the reliability of the result is improved.

3. According to the real-time binocular pupil inspection system and the real-time binocular pupil inspection method, pupil positioning and pupil extraction are respectively carried out through the lightweight target detection model Yolo and the deep learning semantic segmentation model FastSCNN, the robustness of pupil positioning and extraction is improved, and the problem of reflection of infrared lamps on the pupil surface is solved to a certain extent.

Drawings

Fig. 1 is a schematic diagram of a real-time binocular pupil inspection system according to an embodiment of the present invention.

Fig. 2 is a schematic block diagram of a real-time binocular pupil inspection system according to an embodiment of the present invention.

Detailed Description

The objects and effects of the present invention will become more apparent from the following detailed description of the preferred embodiments and the accompanying drawings, it being understood that the specific embodiments described herein are merely illustrative of the invention and not limiting thereof.

As shown in fig. 1, the real-time binocular pupil inspection system of the present embodiment includes an imaging module, an image acquisition module, a pupil position detection module, a pupil extraction module, a binocular distance measurement module, a pupil measurement module, a light source control module, and a data statistics module.

The imaging module comprises a light source and two groups of mutually isolated binocular infrared cameras. In this embodiment, the light source is an LED lamp bead. Each binocular infrared camera is provided with two lenses and an infrared lamp, the model of the camera is JSK-S8130X2, and the fixed focus is 10cm. The focusing distance of each camera is 50-150 mm, the wavelength of the infrared lamp is 850nm, the distance between optical centers is 25mm, the width of the lens is 14mm, and the viewing angle is 86 degrees. The LED lamp can be selected from any lamp beads supporting adjustable brightness, the brightness range is required to be 5-50 lux, and the LED lamp is connected through a Type-C interface. The design is to provide reliable illumination brightness, and the LED lamp consists of a row of 7 lamp beads. The light source is controlled by a light source control module to control the brightness and duration of the light source. If the left lamp bead is controlled to emit 100lux brightness, the lamp bead is continuously lighted for 3 seconds.

The binocular infrared camera inputs the acquired images into an infrared image sequence in pairs. In this embodiment, the camera and the fixed base of the LED and the LED partition are printed using a 3D printer. In the embodiment, the LED lamp is controlled by the LED control module, and an instruction is sent to the LED equipment through serial communication, so that the intensity, the color and the duration of illumination can be controlled. The serial communication protocol agrees that the frame contains a frame header, a function code, an RGB value, a lamp lighting duration and a frame tail. The color standard supported is RGB24, i.e. 24 bits are used to describe one color. The duration time ranges from 0 to 120 seconds, and the brightness adjustable range is 0 to 1000lux.

The two binocular infrared cameras shoot the left eye and the right eye of a tester in real time, and for each eye, the corresponding binocular infrared camera can acquire two images at each moment, and the sampling rate of the camera is 30 frames per second. The acquired images are input into the infrared image sequence in groups of two.

As shown in fig. 1, the image acquisition module, the pupil position detection module, the pupil extraction module, the binocular distance measurement module, the pupil measurement module, the light source control module and the data statistics module are integrated in the base of the binocular pupil inspection system.

The images acquired by the two binocular infrared cameras are input into an image acquisition module, and the image acquisition module uses histogram equalization to preprocess the images, so that the contrast ratio of the images is enhanced. And extracting infrared images by taking two images as units in the infrared image sequence output by the image acquisition module, wherein the two extracted images are left eye images, and the two extracted images are right eye images, so that the cycle is performed.

The pupil position detection module is used for monitoring the pupil positions of the left eye and the right eye in real time by training a lightweight target detection model Yolo. The pupil position includes a pixel representation of the pixel coordinates (x, y) of the pupil center, pupil region width w, and height h. In this embodiment, 100 pupil images are acquired using an infrared camera and the pupil position and binary mask are labeled for use as a dataset to train a lightweight object detection model Yolo. The average time of the training lightweight target detection model Yolo to the inference of each frame of image is 6ms, which can reach more than 140FPS, and the real-time requirement of pupil position detection by a pupil meter is met.

And after the pupil position detection module detects the pupil position, the pupil extraction module cuts out a pupil local image based on the pupil position, and inputs the pupil local image into the trained FastSCNN model to extract a pupil region. In this embodiment, the average time spent for extracting the pupil area by the trained FastSCNN model is 2ms, which can reach more than 500FPS, and meets the requirement of pupil extraction instantaneity. The pupil region in the image can be obtained through the cascade pupil position detection module and the pupil extraction module, and the calculation mode is as follows:

p＝M _FastSCNN (M _yolo (I))

wherein I is an input infrared image, M _yolo Output pupil position (x, y, w, h), M _FastSCNN Based on this position, an actual pupil area image p is extracted, the width and height of which are W, H in pixels.

When the pupil is positioned in the image, blink detection is needed to determine whether the eyes of the person in the current image are semi-closed, i.e. intermediate state between fully open and fully closed, onlyThe subsequent sub-module is executed with the eyes fully open, otherwise the next frame of image is skipped to be directly processed. The image of pupil area is binarized, the gray scale of pixel value is greater than 128 and is marked as 1, otherwise, is marked as 0, the binarization result of the area is summed up, and compared with the result of the previous frame image, when the variation amplitude T exceeds T _max When the current frame is judged to be blinking, i.e. invalid, 80% of the time, the processing of the remaining sub-modules is skipped and the next frame of infrared image is processed. The calculation method of the variation amplitude T comprises the following steps:

T＝(B(p _cur )-B(p _prev ))/B(p _prev )

wherein B represents binarizing the input image, counting 1, and p _prec 、p _cur And extracting actual pupil area images for the previous frame and the current frame respectively.

The binocular distance measuring module is used for calculating the actual distance between each pupil and the corresponding binocular infrared camera. Each binocular infrared camera shoots two images, which are respectively marked as an infrared image 1 and an infrared image 2 from left to right. For the left eye, infrared image 2 is taken as a reference and for the right eye, infrared image 1 is taken as a reference, whereby d is calculated.

The specific distance calculation formula of binocular ranging is as follows:

d＝f·b/s

where f is the camera focal length (in millimeters), b is the distance between the optical centers of the two lenses of the binocular infrared camera (in millimeters), and s is the parallax distance (in pixels) of the pixel coordinates of the centers of the pupils of the left and right eyes. Thereby calculating the actual distance d (in millimeters) of the pupil from the camera. The distance measurement error is less than or equal to 2 percent and the pupil diameter error calculated based on a matrix operation method is less than or equal to 3 percent at the distance of 10 cm-20 cm.

The pupil measuring and calculating module comprises a pupil diameter calculating sub-module, a pupil shape judging sub-module and a pupil sight tracking sub-module. According to the pupil area width w (the unit is a pixel), the actual pupil size (the unit is a millimeter) can be obtained through a matrix operation method, and the calculation mode is as follows:

size＝d·M·W

where W is the width of the pupil area image and M is the internal reference matrix of the camera.

Fitting the extracted pupil region by using a least square algorithm of ellipse fitting, wherein the calculation method comprises the following steps:

e＝F(p)

wherein p is a binary image of an input pupil area image, and F represents an ellipse parameter e which is estimated on pixel point distribution of the binary image area by using an ellipse fitting algorithm, wherein the e comprises a position, an angle, a long axis and a short axis of an ellipse. The shape of one pupil area is represented using the formula (a-b)/a, where a, b are the major and minor axes of the fitted ellipse e, respectively, in pixels.

The pupil sight tracking sub-module is used for calculating the actual movement distance of the pupil in the test process through a phase correlation algorithm.

The data statistics module is used for recording the real-time pupil size, shape and pupil position information into a database, SQLite and Excel files and video data of an inspection process, analyzing the pupil size change range and pupil change rate, drawing a pupil change curve and a pupil position distribution scatter diagram, and displaying the statistics data on a display screen in real time.

The invention also provides a real-time binocular pupil inspection method, which is realized based on the binocular pupil inspection system, and comprises the following steps:

step one: the method comprises the steps of starting a light source to stimulate eyes of a tester, shooting left eyes and right eyes of the tester in real time by adopting two independent binocular infrared cameras respectively, enabling each binocular infrared camera to simultaneously acquire two images of corresponding eyes at each moment, and sending two images into an image acquisition module in a group of two images;

step two: preprocessing the image by using histogram equalization by an image acquisition module, and enhancing the contrast of the image;

step three: inputting the two-by-two images processed by the image acquisition module into a trained lightweight target detection model Yolo by the pupil position detection module, detecting pupil positions of the left eye and the right eye of a tester in real time, performing blink detection, and if the current frame is judged to be in a blink state, skipping other steps, and executing the first step; when the fact that the current pupil position is not located in the edge area of the infrared image is detected, executing the fourth step;

step four: extracting an exit pupil region near the detected pupil position by a pupil extraction module by adopting a trained deep learning semantic segmentation model FastSCNN;

step five: calculating the actual distance from each pupil to the corresponding binocular infrared camera by the binocular distance measuring module, and executing the step six when the actual distance is in a preset distance range;

step six: measuring the actual diameter of the pupil by a pupil measuring module, performing ellipse fitting on the extracted pupil area, and calculating the actual movement distance of the pupil in the test process;

it will be appreciated by persons skilled in the art that the foregoing description is a preferred embodiment of the invention, and is not intended to limit the invention, but rather to limit the invention to the specific embodiments described, and that modifications may be made to the technical solutions described in the foregoing embodiments, or equivalents may be substituted for elements thereof, for the purposes of those skilled in the art. Modifications, equivalents, and alternatives falling within the spirit and principles of the invention are intended to be included within the scope of the invention.

Claims (10)

1. The real-time binocular pupil inspection system is characterized by comprising an imaging module, an image acquisition module, a pupil position detection module, a pupil extraction module, a binocular distance measurement module and a pupil measurement module;

the imaging module comprises a light source and two groups of mutually isolated binocular infrared cameras, wherein the light source is used for providing a light source to stimulate a through hole of a tester during testing; the two groups of mutually isolated binocular infrared cameras are used for shooting the left eye and the right eye of a tester in real time, the binocular infrared camera corresponding to each eye can acquire two images at each moment, and the acquired images are input into an infrared image sequence in pairs;

the image acquisition module receives infrared image sequences acquired by two binocular infrared cameras, and preprocessing the images by using histogram equalization to enhance the contrast of the images;

the pupil position detection module inputs the images which are processed by the image acquisition module into a trained lightweight target detection model Yolo, and detects pupil positions of the left eye and the right eye of a tester in real time;

the pupil extraction module adopts a trained deep learning semantic segmentation model FastSCNN to extract an exit pupil area near the detected pupil position;

the binocular distance measuring module is used for calculating the actual distance between each pupil and the corresponding binocular infrared camera;

the pupil measuring and calculating module comprises a pupil diameter calculating sub-module, a pupil shape judging sub-module and a pupil sight tracking sub-module; the pupil diameter calculation submodule is used for calculating the actual diameter of the pupil; the pupil shape judging submodule is used for carrying out ellipse fitting on the extracted pupil area and determining parameters of the ellipse; the pupil sight tracking sub-module is used for calculating the actual movement distance of the pupil in the test process and performing blink detection.

2. The real-time binocular pupil inspection system of claim 1, wherein the pupil location comprises a pixel representation of the pixel coordinates of the pupil center, pupil region width and height.

3. The real-time binocular pupil inspection system of claim 1, wherein the calculation formula for calculating the actual distance of each pupil to the corresponding binocular infrared camera is as follows:

d＝f·b/s

wherein f is the focal length of the binocular infrared camera, b is the optical center distance between two lenses of the binocular infrared camera, s is the parallax distance of the pixel coordinates of the centers of pupils of the left eye and the right eye, and the unit is a pixel.

4. A real-time binocular pupil inspection system as claimed in claim 3, wherein the specific process of calculating the actual diameter size of the pupil by the pupil diameter calculation sub-module comprises:

obtaining the actual diameter of the pupil through matrix operation size=d.M.P according to the width w (the unit is a pixel), wherein P is the pixel difference of the left boundary point and the right boundary point on the same horizontal line of the pupil area in the horizontal direction of the pixel coordinates; m is an internal reference matrix of the camera.

5. The real-time binocular pupil inspection system of claim 1, wherein the pupil shape determination sub-module determines the ellipse parameters by minimizing the point-to-ellipse distance using a least squares-based ellipse fitting method when performing ellipse fitting of the extracted pupil area.

6. The real-time binocular pupil inspection system of claim 1, wherein the pupil gaze tracking sub-module calculates the actual distance of movement of the pupil by a phase correlation algorithm.

7. The real-time binocular pupil inspection system of claim 1, further comprising a light source control module for controlling the brightness and duration of the light source.

8. The real-time binocular pupil inspection system of claim 1, wherein the light source is an LED.

9. The real-time binocular pupil inspection system of claim 1, further comprising a data statistics module for recording real-time pupil size, shape, pupil location information to a database.

10. A real-time binocular pupil inspection method, characterized in that the method is implemented based on the binocular pupil inspection system of any one of claims 1 to 9, the method comprising the steps of:

step one: the method comprises the steps of starting a light source to stimulate eyes of a tester, shooting left eyes and right eyes of the tester in real time by adopting two independent binocular infrared cameras respectively, enabling each binocular infrared camera to simultaneously acquire two images of corresponding eyes at each moment, and sending two images into an image acquisition module in a group of two images;

step two: preprocessing an image by using histogram equalization by the image acquisition module, and enhancing the contrast of the image;

step three: inputting the images processed by the image acquisition module into a trained lightweight target detection model Yolo by the pupil position detection module, detecting pupil positions of the left eye and the right eye of a tester in real time, performing blink detection, and if the current frame is judged to be in a blink state, skipping other steps, and executing the first step; when the fact that the current pupil position is not located in the edge area of the infrared image is detected, executing the fourth step;

step four: extracting an exit pupil region near the detected pupil position by the pupil extraction module by adopting a trained deep learning semantic segmentation model FastSCNN;

step five: calculating the actual distance from each pupil to the corresponding binocular infrared camera by the binocular distance measuring module, and executing the step six when the actual distance is in a preset distance range;

step six: and measuring the actual diameter of the pupil by the pupil measuring and calculating module, performing ellipse fitting on the extracted pupil area, and calculating the actual movement distance of the pupil in the test process.