WO2022077800A1 - 眼底相机及眼底图像全自动拍摄方法 - Google Patents

眼底相机及眼底图像全自动拍摄方法 Download PDF

Info

Publication number
WO2022077800A1
WO2022077800A1 PCT/CN2021/073875 CN2021073875W WO2022077800A1 WO 2022077800 A1 WO2022077800 A1 WO 2022077800A1 CN 2021073875 W CN2021073875 W CN 2021073875W WO 2022077800 A1 WO2022077800 A1 WO 2022077800A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
fundus
lens
pupil
focal length
Prior art date
Application number
PCT/CN2021/073875
Other languages
English (en)
French (fr)
Inventor
胡丁山
庞新强
姜欣
郭韩越
任文斌
常献刚
王鹏
魏宇博
和超
张大磊
Original Assignee
上海鹰瞳医疗科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海鹰瞳医疗科技有限公司 filed Critical 上海鹰瞳医疗科技有限公司
Priority to EP21878859.4A priority Critical patent/EP4230112A1/en
Priority to US18/031,513 priority patent/US20230404401A1/en
Priority to JP2023519780A priority patent/JP2023547595A/ja
Publication of WO2022077800A1 publication Critical patent/WO2022077800A1/zh

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/12—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for looking at the eye fundus, e.g. ophthalmoscopes
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/0008—Apparatus for testing the eyes; Instruments for examining the eyes provided with illuminating means
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/14—Arrangements specially adapted for eye photography
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/14—Arrangements specially adapted for eye photography
    • A61B3/15—Arrangements specially adapted for eye photography with means for aligning, spacing or blocking spurious reflection ; with means for relaxing
    • A61B3/152—Arrangements specially adapted for eye photography with means for aligning, spacing or blocking spurious reflection ; with means for relaxing for aligning
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2560/00—Constructional details of operational features of apparatus; Accessories for medical measuring apparatus
    • A61B2560/04—Constructional details of apparatus
    • A61B2560/0431—Portable apparatus, e.g. comprising a handle or case

Definitions

  • the invention relates to the field of ophthalmic instruments, in particular to a fundus camera and an automatic fundus image capturing method.
  • the retina is the only tissue in the human body that can directly observe capillaries and nerves. By observing the retina, not only eye health problems but also systemic diseases such as diabetes complications and high blood pressure can be found.
  • a fundus camera is a specialized device used to photograph the retina.
  • Existing fundus cameras can automatically capture fundus images, and the process of automatic shooting mainly involves automatically aligning the main lens with the pupil, automatically adjusting the axial distance (working distance) between the lens and the pupil, and automatically adjusting the focal length.
  • the camera is equipped with a main camera, an auxiliary camera and many auxiliary optics.
  • the main camera is installed on a platform that can move in X, Y, and Z directions to capture the fundus; the auxiliary camera is installed near the main camera to capture the face. It is mainly used to search the eye and realize automatic pupil alignment; auxiliary optics are used for focusing, adjusting the working distance, etc.
  • the existing fundus camera requires complex and expensive hardware modules, and is also complicated to use, which hinders the popularization of fundus cameras.
  • the present invention provides an automatic fundus image capturing method, comprising:
  • a fundus image is captured using the capture focal length at the working distance.
  • the method before moving the lens of the fundus camera to align with the pupil, the method further includes: detecting whether the motion component, the lighting component and the focusing component of the fundus camera are normal.
  • detecting whether the motion components, lighting components and focusing components of the fundus camera are normal specifically includes:
  • Control the movement component to adjust the position of the lens, and detect whether the lens can move to the position where each positioning component is located;
  • the moving assembly is controlled to move the lens to a set position, the lighting assembly is turned on, and the focusing assembly is controlled to be adjusted to a first focal length, and a first image is obtained by shooting;
  • control the moving assembly to adjust the lens to the set depth position, control the focusing assembly to adjust to the second focal length, and capture a second image
  • Whether the imaging function is normal is determined according to the image characteristics of the object in the second image.
  • the method before moving the lens of the fundus camera to align with the pupil, the method further includes: detecting whether the head of the human body fits with the face-sticking component of the fundus camera.
  • the face-sticking component for detecting whether the human head fits the fundus camera specifically includes:
  • the second image it is determined whether the head of the human body fits the surface patch assembly.
  • using the image to determine the working distance specifically includes:
  • using the fundus image to determine the focal length specifically includes:
  • the shooting focal length is determined according to the sharpness of the optic disc area.
  • using the shooting focal length to capture the fundus image at the working distance specifically includes:
  • the multiple fundus images are fused into one fundus image.
  • using the shooting focal length to capture the fundus image at the working distance specifically includes:
  • a fundus image is synthesized using a plurality of the high-quality regions.
  • the present invention provides an electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein, the memory stores instructions executable by the one processor, The instructions are executed by the at least one processor, so that the at least one processor executes the above-mentioned fully automatic fundus image capturing method.
  • the present invention provides a fundus camera, comprising: a face-mounted component, a motion component, a focusing component, an illumination component, a lens, and at least one processor; and a memory communicatively connected to the at least one processor; wherein, the The memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor to cause the at least one processor to execute the above-described fully automatic fundus image capturing method.
  • the fundus camera can automatically align the main lens with the pupil, automatically adjust the working distance, and automatically adjust the focal length.
  • the image does not require auxiliary cameras and auxiliary optical devices, which reduces the complexity and difficulty of use of the hardware, allows users to independently capture fundus images, and promotes the popularization of fundus cameras.
  • FIG. 1 is a structural diagram of a fundus camera in an embodiment of the present invention.
  • FIG. 2 is a schematic diagram of a surface sticker assembly of a fundus camera in an embodiment of the present invention
  • FIG. 3 is a schematic diagram of a lens and a positioning assembly
  • FIG. 4 is a flowchart of an automatic fundus image capturing method according to an embodiment of the present invention.
  • Figure 5 is a schematic diagram of pupil labeling
  • FIG. 6 is a flowchart of a preferred fully automatic method for capturing fundus images in an embodiment of the present invention
  • Fig. 7 is the schematic diagram that pupil is larger than illumination beam
  • Fig. 8 is the schematic diagram that pupil is smaller than illumination beam
  • FIG. 9 is a schematic diagram of capturing a fundus image when the pupil is smaller than the illumination beam
  • Fig. 10 is the imaging of corneal reflected illumination beam
  • 11 is a schematic diagram of the distance between the lens barrel and the eyeball
  • Figure 12 is a schematic diagram of spot labeling
  • Fig. 13 is the imaging of corneal reflected illumination beam when the working distance is reached
  • Figure 14 is a schematic diagram of video disc labeling
  • 15 is a schematic diagram of moving a lens position according to a light spot when capturing a fundus image
  • 16 is a schematic diagram of two fundus images with unavailable areas
  • 17 is a schematic diagram of a synthesis method of a fundus image
  • Figure 18 is a structural diagram of a lighting lamp
  • FIG. 19 is a schematic diagram of imaging reflected light of illumination when detecting the state of the camera.
  • Fig. 20 is the imaging schematic diagram of the convex part of the face-to-face assembly when detecting the state of the camera;
  • 21 is a schematic diagram of imaging of the convex portion of the surface mount assembly provided with the target when the state of the camera is detected;
  • FIG. 22 is an image of the area between the eyes collected when the use state of the person being photographed is detected.
  • the terms “installed”, “connected” and “connected” should be understood in a broad sense, unless otherwise expressly specified and limited, for example, it may be a fixed connection or a detachable connection connection, or integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal connection of two components, which can be a wireless connection or a wired connection connect.
  • installed installed
  • “connected” and “connected” should be understood in a broad sense, unless otherwise expressly specified and limited, for example, it may be a fixed connection or a detachable connection connection, or integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal connection of two components, which can be a wireless connection or a wired connection connect.
  • Figure 1 shows a fully automatic portable self-portrait fundus camera.
  • the camera includes a face sticker component 01, a motion component, a positioning component 03 and a lens barrel 1.
  • the lens barrel 1 is provided with an illumination component, a focusing component, and a lens (eye-connecting component). Objective lens), optical lens group and imaging detector 10, etc., the internal structure of lens barrel 1 can refer to Chinese patent document CN111134616A.
  • the actual product also includes the housing, and the motion components and the lens barrel 1 are located inside the housing.
  • the face sticker assembly 01 is sealingly connected to the front of the housing, and the face sticker assembly includes a face sticker body and a window through hole formed on the face sticker body for accommodating the eye of the subject when the eye is attached.
  • the face sticker assembly 01 is used as a component that contacts the eye of the subject, and the lens barrel 1 collects the retinal image of the subject's eye through the through hole of the face sticker assembly 01 .
  • the side of the face sticker body facing away from the lens barrel 1 is configured in a shape that conforms to the facial contour around the eyes of the subject.
  • the face patch assembly 01 is inwardly formed in a concave shape to suit the arc shape of the human head, and the size of the through hole can accommodate both eyes at least when the subject's eyes fit the assembly.
  • FIG. 2 shows the inward-facing side of the surface sticker assembly 01 .
  • the lens can be aimed at this part and take an image.
  • a more preferred solution is to set a pattern or a simple figure on the raised portion 012 as a target.
  • This specific position has various uses, including detecting whether the camera's lighting components and focusing components are normal, detecting whether the subject's eyes are correctly attached to the face sticker component 01, etc., which will be described in detail below.
  • the motion component is used to control the movement of the lens barrel 1 in the three-dimensional space. Taking the coordinate system in FIG. 1 as an example, it can move on the three axes of X, Y, and Z in the figure. It should be noted that, when the lens barrel 1 moves to the limit position in the Z direction, the end portion will not protrude out of the surface sticker assembly 01 .
  • the motion component includes three track components, the first group of tracks 021 is used to control the movement of the lens barrel 1 on the X axis, and the second group of tracks 022 is used to control the movement of the lens barrel 1 on the Y axis , The third group of tracks not shown in the figure is used to control the movement of the lens barrel 1 on the Z axis.
  • the lens barrel 1 and the second group of rails 022 are arranged on a platform (base), the first group of rails 021 can drive the overall movement of the base, and the third group of rails can drive the base and the first group of rails 021 to move , so that the whole is close to or away from the surface-mounted component 01.
  • the positioning assembly 03 is used to detect the movement of the lens barrel 1 .
  • the positioning assembly 03 may be an electromagnetic sensor, which senses the movement of the lens barrel 1 to the position where the positioning assembly 03 is located according to the electromagnetic induction signal.
  • three positioning assemblies 03 are provided, two of which are arranged on both sides of the movable base to detect the movement of the lens barrel 1 on the X axis, and the third positioning assembly is provided On the base, the movement of the lens barrel 1 on the Y axis is detected, that is, the positioning assembly 03 is used to detect the movement of the lens barrel 1 in the XY plane.
  • the illumination component for imaging, the focusing component, the objective lens, the optical lens group and the imaging detector are integrated in one lens barrel to realize the miniaturization of the optical path structure, reduce the volume of the fundus camera, and improve the portability.
  • the surface mount component of the fundus camera is provided with a window through hole for accommodating the eye of the photographed subject. The user can wear the fundus camera by himself, and place the eye at the position of the window through hole, and the motion component drives the lens barrel in the window through hole range. It searches the pupil and adjusts the working distance to capture the fundus image. This solution reduces the complexity and difficulty of using the fundus camera hardware, enables users to take the fundus image independently, and promotes the popularization of the fundus camera.
  • Embodiments of the present invention provide a fully automatic method for capturing a fundus image, which can be performed by a fundus camera itself, or performed by an electronic device such as a computer or a server (as a control method). As shown in Figure 4, the method includes the following steps:
  • the steps of detecting the camera state and the user's use state may also be performed.
  • the method may further include:
  • S100 detecting whether the motion components, lighting components and focusing components of the fundus camera are normal. This step is optional and can be performed while the fundus camera is turned on. If an abnormality of a component is detected, the subsequent shooting operation will be terminated and a corresponding abnormality prompt will be given.
  • S200 detecting whether the head of the human body fits the face-sticking component of the fundus camera. This step is an optional operation. If it is detected that the human head does not fit the face-mounting component of the fundus camera, the user can be prompted through the voice module to guide the user to wear the fundus camera correctly.
  • an embodiment of the present invention provides a fundus camera detection method, which can be performed by the fundus camera itself, as a self-checking method, or performed by an electronic device such as a computer or a server, as a product detection method method, the method includes the following steps:
  • Step S1 control the moving component to adjust the position of the lens, and detect whether the lens can move to the position where each positioning component is located.
  • This method is suitable to be executed when the fundus camera is just started.
  • the lens (according to the above embodiment, the lens and the lens barrel are integrally provided, and the moving lens barrel is the moving lens) is moved to the initial position.
  • the moving component adjusts the position of the lens, and detects whether the lens can move to the position where the three positioning components are located. If it can move to these positions, it is considered that the function of the moving component is normal, and step S2 can be performed; otherwise, step S6 is performed.
  • Step S1 may be referred to as a step of detecting the movement of the XY axis of the moving component.
  • This step is to detect whether the focusing assembly and the lighting assembly function normally. In theory, there is no need to limit the lens to a certain position, so there are various options for setting the position described in this step.
  • the external environment is uncertain, for example, it may be a relatively bright environment. If the external environment is relatively bright when the first image is captured in this step, the content of the image may be disturbed.
  • this step moves the lens to a specific part of the surface mount component (such as the above-mentioned protrusion), and captures as little external environment as possible, and the proportion of the surface mount component in the image is larger than that of the outside proportion of the environment.
  • a specific part of the surface mount component such as the above-mentioned protrusion
  • the proportion of the surface mount component in the image is larger than that of the outside proportion of the environment.
  • Fig. 18 shows the structure of a lighting lamp in a lens barrel, and four lamp beads are arranged on a ring structure. Turn on these four lamp beads and use the first focal length for imaging, and an image as shown in Figure 19 is expected to be obtained.
  • the set first focal length can make the illumination assembly image but not the surface mount assembly. Therefore, only the lighting component may exist in the first image without objects such as the protrusion of the surface sticking component, which improves the accuracy of image recognition in the subsequent steps.
  • S3 Determine whether the focusing assembly and the lighting assembly are normal according to the image characteristics of the lighting assembly in the first image.
  • using the set focal length should yield the image shown in Figure 19, which has distinct features depending on the actual shape of the lighting assembly.
  • there should be 4 independent and clear dots in the first image which are the imaging results of the above 4 lamp beads. If the focal length adjusted at this time is not the above-mentioned first focal length, the dots in the image will become larger and blurred, or become smaller; if the lighting assembly is not turned on, no shape will appear in the image.
  • Step S4 By recognizing the first image through a machine vision algorithm or a neural network algorithm, it can be identified whether there are features that meet expectations in the image. If it is determined that the focusing assembly and the lighting assembly are normal, step S4 is performed, otherwise, step S6 is performed. Steps S2-S3 may be referred to as focusing assembly and lighting assembly detection steps.
  • step S4 controlling the moving component to adjust the lens to the set depth position, controlling the focusing component to adjust to the second focal length, and shooting to obtain a second image.
  • a known object needs to be imaged, and as a preferred embodiment, the raised portion of the face sticker assembly is used as the known object.
  • step S2 first align the lens on the XY plane with the raised part of the surface mount assembly.
  • step S2 has already been aligned with this part, and this step does not need to be adjusted; in other embodiments, if step S2 If this part is not aligned, adjust it in this step.
  • This step requires adjusting the depth, that is, adjusting the position of the lens on the Z axis, which can be understood as adjusting the shooting distance of a known object, and then setting the focal length.
  • the focal length at this time is different from the focal length used in step S2, and the focal length in this step should be suitable for the current lens position (depth position), and it is expected to get a picture as shown in Figure 20. image.
  • the set second focal length can make the surface mount assembly image but not the lighting assembly.
  • the object to be photographed such as the raised portion of the surface sticker assembly, may exist in the second image, and the image of the lighting assembly will not appear, thereby improving the accuracy of image recognition in the subsequent steps.
  • S5 Determine whether the imaging function is normal according to the image features of the object in the second image.
  • the second image is an image captured by the moving component when the XY axis movement, the lighting component and the focusing component are all normal. The purpose of this step is to detect whether the moving component moves normally on the Z axis. If the moving component can Adjust the lens to the set depth position, and the second image taken should be able to show a clear object, such as the raised part of the face sticker assembly shown in Figure 20.
  • Step S6 is performed. Steps S4-S5 may be referred to as Z-axis motion detection steps of the motion component.
  • a voice module or an information display module can be set in the fundus camera to broadcast or display the corresponding fault information to the user.
  • the positioning component is used to verify whether the moving component can normally adjust the position of the lens; after confirming that the moving component is normal, the focal length is adjusted to make the lighting component image, and the collected image can be determined by judging Whether the focusing component and lighting component are normal; finally, adjust the depth of the lens through the motion component, adjust the focal length to image the object, and judge the characteristics of the object in the image to verify whether the motion component can adjust the depth of the lens normally, thus automatically Determine whether the various important parts of the fundus camera work properly.
  • This solution can self-check the working status of the equipment in a remote unattended environment, thereby improving the convenience of taking fundus photos and promoting the popularization of fundus cameras.
  • a target is provided on the raised part of the face-sticking assembly, that is, the above-mentioned object to be photographed is a target on the set part of the face-sticking assembly.
  • the specific content of the target is not limited, and is one or more A well-defined pattern or shape is possible.
  • the obtained second image is shown in FIG. 21, which includes a circular target 81.
  • Step S5 specifically includes:
  • an embodiment of the present invention provides a method for detecting a use state of a fundus camera, which is used to detect whether the user is wearing the fundus camera in the above embodiment correctly.
  • the method can be performed by the fundus camera itself, and as a self-checking method, it can also be performed by an electronic device such as a computer or a server.
  • the method is suitable to be executed after it is determined that the important components of the camera are normal according to the above detection method, and the method includes the following steps:
  • the lens collects the image of the external environment through the through hole 011 as shown in Figure 2, and it should be avoided that the face mount component blocks the lens (the face mount component is not within the imaging range).
  • the face mount component blocks the lens (the face mount component is not within the imaging range).
  • the lighting assembly needs to be kept off, that is, the beam is not directed outward through the lens.
  • the focal length used for the captured image can be a fixed value, and the imaging plane can be roughly set on the surface of the human body.
  • step S2 judging whether the brightness of the first image reaches a set standard. If the eye of the person being photographed is in close contact with the face-sticking assembly 01 and there is no large gap around it, the captured first image should be very dark. Here, the brightness of the first image is judged first, and if the brightness reaches the set standard, step S3 is performed, otherwise, step S6 is performed.
  • the brightness value can be calculated according to the pixel value of the image, and then compared with the threshold value; the neural network algorithm can also be used to train the neural network with images of different brightness in advance. , so that it has the ability to classify or predict the brightness of the image.
  • the neural network is used to identify the first image, and the identification result about the brightness is output.
  • the first image is converted into a grayscale image, and then the brightness of the grayscale image is identified.
  • the lighting component is turned on, and the second image captured by the lens through the window of the surface mount component is acquired.
  • the state of the lens and the subject has not changed, but only the illumination light source is turned on, and the lens is illuminated outwards.
  • the illumination beam irradiates the subject's eyes or skin and is reflected.
  • the position of the lens is set to be aligned with the center of the window of the surface-mounted component, and the light source used is infrared light. If the human head is attached to the surface-mounted component, the lens is aimed at the eyes The area in between can be collected as shown in Figure 22.
  • S4 determine whether the head of the human body fits the surface sticking component. If the subject's head is attached to the face-mounted component, since the human skin will reflect the illumination beam, an obvious light spot will appear in the image as shown in Figure 22, and the characteristics of human skin will appear around the light spot. By judging the image Whether it has the characteristics of a brighter center and a darker edge can determine whether the human head fits the surface patch component.
  • Steps S1-S2 the camera is placed in a dark room, or the surface mount component is covered by other objects, the brightness of the first image will also be determined to meet the set standard, which requires further execution Steps S3-S4 are for judgment. If there is no object attached to the surface mount component, there will be no light spots in the second image collected; if other objects cover the surface mount component, light spots will appear in the second image, but due to different materials and surface shapes, for the illumination beam The reflection situation is different from that of the human body, so whether it is a human body can be judged by the characteristics of the light spot.
  • the lens may also be aimed at other positions when the first and second images are collected, such as at the eyeball, and in step S4, the eyeball feature may be identified in the image to determine whether it is a human body.
  • step S4 may first determine whether the brightness of the second image reaches a set standard, and similar to identifying the brightness of the first image, the second image may be converted into a grayscale image, and then the brightness value may be calculated, or Recognition using neural network. If there is a gap between the surface mount component and the human body at this time and light leakage occurs, the brightness of the second image will be different from the brightness of the second image when only the light source of the camera itself is illuminated due to the influence of ambient light. After the light leakage is excluded, it is then judged whether the features in the second image conform to the features of human skin.
  • Step S5 is performed when it is determined that the human head fits the face-sticking component, otherwise, step S6 is performed.
  • S5 start capturing the fundus image. Specifically, it is necessary to automatically find the pupil, adjust the working distance, and adjust the focal length to set the imaging plane on the fundus, and finally capture the fundus image.
  • a voice module may be set in the fundus camera to prompt the user how to properly wear the fundus camera, etc., and then return to step S1 to make a new judgment.
  • an image is collected when the lighting assembly is turned off, and it can be preliminarily determined whether the face-sticking assembly is well covered by the object through the brightness of the image, and then the image is collected when the lighting assembly is turned on. , and further determine whether the covering is a human body through the image features, thereby automatically determining whether the subject is wearing the fundus camera correctly and using the fundus camera in a suitable environment.
  • This solution can automatically trigger the fundus camera to take fundus photos. Manual intervention is required to trigger shooting, and no professional manipulation is required, thereby improving the convenience of taking fundus photos and promoting the popularization of fundus cameras.
  • an embodiment of the present invention provides a method for automatically aligning a lens of a fundus camera, which can be performed by the fundus camera itself, or performed by an electronic device such as a computer or a server (as a control method). It includes the following steps:
  • S1 identify the image collected by the lens of the fundus camera, and determine whether there is a pupil in it. Specifically, after the user wears the above-mentioned fundus camera, the system will continuously (such as frame by frame) capture images of the pupil. If the pupil can be identified in the image, it means that the pupil is already within the imaging range. In this case, fine-tuning is performed to make The lens is perfectly aligned with the pupil to shoot. If the pupil cannot be identified in the image, it means that the position of the lens and the pupil has a large deviation. The reason may be that the initial position of the lens is not suitable, or the user's wearing method is not standard, etc.
  • a large number of photos of pupils are collected. These photos are images collected by different people, at different directions and distances from the eyepiece objective of the above-mentioned fundus camera, and at different times. Then, the pupils in each image are marked to obtain training data for training the neural network. Use these labeled data to train a neural network model (such as the YOLO network). After training, the recognition result of the neural network model includes a detection box, which is used to characterize the position and size of the pupil in the image.
  • a neural network model such as the YOLO network
  • a square frame 51 is used to mark the pupil in the training data, and the recognition result of the trained neural network model will also be a square detection frame.
  • a circular frame may also be used for marking, or other similar marking methods are feasible.
  • step S2 it is only necessary to identify whether there is a pupil in the image, if there is no pupil in the image, step S2 is performed, otherwise, step S3 is performed.
  • the user will be prompted to adjust the wearing state; if the pupil is searched, it will be further judged if the user's eyes are far away from the lens, beyond the range that the motion component can move, such as judging the lens Whether the moving distance exceeds the moving threshold, when the moving distance exceeds the moving threshold, the user is prompted to move the head slightly in the face sticker component to adapt to the moving range of the lens. Then continue searching, and execute step S3 when the moving distance does not exceed the moving threshold.
  • S3 determine whether the pupil in the image meets the set condition.
  • setting conditions can be set, such as conditions related to size, conditions related to shape, and the like.
  • the set condition includes a size threshold, and it is determined whether the pupil size in the image is larger than the size threshold.
  • the pupil size in the image is larger than the size threshold, it is determined that there is a pupil that meets the set condition; otherwise, the user is prompted. Close your eyes and rest for a while to dilate your pupils before shooting. Because when taking fundus images, it is generally necessary to take pictures of both eyes in sequence. After taking the first eye, the pupil will be narrowed. Therefore, the system will also let the user close their eyes and rest to restore the pupil size.
  • the set condition includes a morphological feature, and it is determined whether the pupil shape in the image conforms to the set morphological feature.
  • Conditioned pupil otherwise prompt the user to open their eyes, try not to blink, etc.
  • the set morphological feature is circular or approximately circular. If the detected pupil does not conform to the preset morphological feature, for example, it may be flat, which is generally caused by the user's eyes not being opened.
  • the above-mentioned neural network model needs to be used for pupil detection, and the recognition result of the neural network model also includes the confidence information of the pupil, that is, the probability value used to indicate that the model determines that the pupil exists in the image .
  • the setting conditions include a confidence threshold, and it is determined whether the confidence information obtained by the neural network model is greater than the confidence threshold. When the confidence information is greater than the confidence threshold, it is determined that there are pupils that meet the set conditions; otherwise, the user is prompted to open their eyes and remove obstacles such as hair.
  • the confidence of the pupil obtained by the neural network model is relatively low, indicating that although there is a pupil in the image, it may be disturbed by other objects. In order to improve the shooting quality, the user is prompted to make adjustments here.
  • Step S4 is performed when the pupil in the image meets the set condition, otherwise, it waits for the user to adjust his state and continues to judge until the set condition is met.
  • Step S4 move the lens of the fundus camera to align the pupil according to the position of the pupil in the image.
  • the lens barrel is moved by the above-mentioned motion components, and the direction and distance of the movement depends on the deviation of the pupil from the lens in the image.
  • the center point of the acquired image as the center point of the lens, and identify the center point of the pupil in the image.
  • the center point of the detection frame may be regarded as the center point of the pupil.
  • a fundus image capturing method is provided.
  • the pupil state in the image it can be automatically determined whether the subject's current pupil state is suitable for capturing the fundus image.
  • the photographer sends out corresponding prompts to adjust their own state.
  • the state is suitable for shooting fundus images
  • the position of the pupil is recognized to perform automatic alignment, and then the shooting is performed, thereby avoiding unusable fundus images.
  • No professional participation is required, enabling users to shoot autonomously.
  • the size of the pupil may be smaller than the size of the annular illumination beam.
  • aligning the pupil with the eyepiece will cause no light to enter the pupil, so the image is captured.
  • the image is black.
  • an embodiment of the present invention provides a preferred method for capturing a fundus image, and the method includes the following steps:
  • FIG. 7 shows a case where the size of the pupil 72 is larger than the size of the annular light beam 71 , and step S52 is executed in this case.
  • Fig. 8 shows the case where the size of the two annular illumination beams is larger than the pupil size.
  • the illumination light source is a complete annular illumination lamp or a light source formed by a plurality of illumination lamps arranged in a ring shape.
  • the inner diameter of the annular beam 71 is larger than that of the pupil. 72 in diameter.
  • Step S53 is executed when the pupil size is smaller than the annular illumination beam size, that is, the situation as shown in FIG. 8 is met.
  • step S54 is executed.
  • S531 determine the edge position of the pupil. Specifically, a machine vision algorithm or the above-mentioned neural network model can be used to obtain the left edge point 721 and the right edge point 722 of the pupil 72 in FIG. 9 .
  • S532 Determine the moving distance according to the edge position of the pupil. Specifically, the moving distance of the moving component can be calculated according to the positional relationship between the position O of the current lens center (image center position) and the left edge point 721 and the right edge point 722 .
  • S533 respectively move the lens in multiple directions according to the determined moving distance, and the determined moving distance makes the edge of the annular illumination beam coincide with the edge of the pupil.
  • the outer edge of the annular beam 71 just coincides with the edge of the pupil 72, so that the part of the annular beam 71 entering the fundus can be located at the edge of the fundus, reducing the impact on the imaging of the central area of the fundus.
  • step S54 fuse the multiple fundus images into one fundus image.
  • the available areas will be extracted from each fundus image, and a complete fundus image will be stitched and fused using these fundus images.
  • step S54 specifically includes:
  • S543a splicing a plurality of effective areas according to the displacement deviation to obtain a spliced fundus image. Further, an image fusion algorithm is used to perform fusion processing at the splicing of each of the effective regions.
  • step S54 specifically includes:
  • a method for capturing a fundus image is provided.
  • the lens of the fundus camera is aligned with the pupil, the size of the pupil in the comparison image and the size of the annular speed of light emitted by the camera itself are first judged. If the pupil size is too small, the illumination beam cannot be illuminated normally.
  • the lens is moved to deviate from the current alignment position so that the annular illumination beam is partially illuminated in the pupil, and fundus images are acquired at multiple offset positions, and finally a fundus image is fused according to multiple fundus images.
  • This scheme The fundus image can be taken when the pupil of the subject is small, without the need for professionals to participate in the shooting process, which reduces the requirements for the pupil state of the subject and improves the shooting efficiency.
  • this embodiment provides a method for adjusting the working distance of a fundus camera, which can be performed by the fundus camera itself, or performed by an electronic device such as a computer or a server (as a control method).
  • the method includes the following steps:
  • control the lens to approach the eyeball and capture an image the image is the imaging of the illumination beam reflected by the cornea.
  • This step is performed according to the solution of the above-mentioned embodiment and is performed when the lens is aligned with the pupil in the XY plane.
  • controlling the lens to approach the eyeball means controlling the lens to move toward the eyeball on the Z axis through the motion component.
  • the light source of the illumination component passes through the optical lens, and the reflected light irradiated on the cornea of the eye is imaged on the cmos, and the result shown in Figure 10 will be obtained.
  • the light source is distributed in a cross shape at The four light balls on the four sides of the lighting assembly also show four light spots in the imaging of this light source.
  • the illumination light source may be in the shape as shown in FIG. 8 , and the captured image will show light spots of corresponding shape or arrangement.
  • the corneal reflected light imaging will change.
  • the position, size, and sharpness of the image are related to the distance between the eyepiece objective and the cornea. The closer the distance is, the greater the angle between the incident light and the normal of the cornea, the heavier the scattering effect of reflection, the larger the spot size, the more divergent, and the lower the brightness.
  • images of a large number of light spots are collected, and these images are images collected by different people at different directions and distances from the eyepiece objective of the above-mentioned fundus camera and at different times. Then, the light spots in each image are marked to obtain training data for training the neural network. Use these labeled data to train a neural network model (such as the YOLO network).
  • the recognition result of the neural network model includes a detection frame, which is used to characterize the position and size of the light spot in the image.
  • a square frame 121 is used to mark the light spot in the training data, and the recognition result of the trained neural network model will also be a square detection frame.
  • a circular frame may also be used for marking, or other similar marking methods are feasible.
  • the set feature can be a feature about size, for example, when the spot size in the image is smaller than the set size, it is determined to meet the set feature; it can also be the disappearance of the spot, such as when the machine vision algorithm or neural network cannot detect the image in the image. It is judged that the light spot conforms to the set characteristics.
  • step S3 If the light spot in the image conforms to the set characteristics, step S3 is performed; otherwise, it returns to step S1 to continue to move the lens and capture images.
  • the imaging of the illumination beam reflected by the cornea is collected and identified, and the distance between the lens and the eyeball is judged and adjusted according to the light spot feature in the image, and it is not necessary to set any setting on the fundus camera.
  • additional optics or hardware it is only necessary to set an appropriate illumination beam to accurately locate the working distance, thereby reducing the cost of the fundus camera and improving the efficiency of working distance adjustment.
  • This embodiment provides a preferred method for adjusting the working distance, and the method includes the following steps:
  • step S2A call the neural network to detect the light spot in the image, and judge whether there is a light spot in the image.
  • step S6A is performed, otherwise, step S3A is performed.
  • step S3A Identify the center point of the light spot in the image, and determine whether the center point of the light spot coincides with the center point of the image.
  • the center of the detection frame obtained by the neural network is regarded as the center of the light spot.
  • the center point of the image is regarded as the center of the lens. If the center point of the image coincides with the center of the light spot, it means that the lens is aligned with the pupil, and step S5A is performed;
  • Detection-adjustment-re-detection is a feedback process.
  • a smooth adjustment algorithm is used here:
  • Adjustment(i-1) represents the displacement of the last lens adjustment
  • Shift(i) represents the offset (the deviation between the pupil center and the image center)
  • Adjustment(i) represents the displacement that needs to be adjusted this time
  • a is A coefficient between 0-1. Because the position of the lens is a two-dimensional coordinate on the XY plane, both Adjustment and Shift are two-dimensional vectors.
  • step S5A is performed.
  • step S5A control the lens to move closer to the eyeball to reduce the distance. Then, returning to step S1A, with the repeated execution of making the lens gradually approach the eyeball, the size of the light spot in the corresponding image changes from large to small.
  • each frame of image can be collected and the above judgment and adjustment can be made accordingly until the image of the disappearance of the light spot is detected.
  • S6A control the lens to continue to move the preset distance in the direction close to the eyeball to reach the working distance.
  • the system while performing the above adjustment process, it will also detect whether the light spot in the image is complete.
  • the light spot is incomplete, such as only half of it, it means that the user is blinking or his eyes are not open. At this time, the system It will prompt the user to open their eyes by voice, try not to blink, etc.
  • the position of the lens is also fine-tuned according to the position of the light spot in the image, thereby keeping the lens aligned with the pupil when adjusting the working distance. It is necessary to set any additional optics or hardware on the fundus camera, and only need to set the appropriate illumination beam to achieve accurate positioning of the working distance and keep the lens aligned with the pupil, thereby reducing the cost of the fundus camera and improving the efficiency of fundus image capture.
  • this embodiment provides a method for adjusting the focal length of the fundus camera, which can be performed by the fundus camera itself, or performed by an electronic device such as a computer or a server (as a control method).
  • the method includes the following steps:
  • S1 adjust the focus and collect fundus images. This step is performed when the lens of the fundus camera is aligned with the pupil and reaches the working distance, and the position of the lens and the eyeball at this time is shown in Figure 13. It should be noted that, in the process of adjusting the lens position and working distance in the above embodiment, when capturing images, it is of course also necessary to set a fixed focal length. If the subject's refraction is normal, when the working distance is adjusted in place, the fundus image can be taken directly. However, in practical application, it is necessary to consider the actual diopter of the person being photographed, so as to set a suitable focal length.
  • infrared light is used for imaging.
  • the light source used for capturing the image is still infrared light.
  • the image collected at this time can basically reflect the characteristics of the fundus, and at least the optic disc can be displayed in the image, so the collected image is called the fundus image.
  • the optic disc in each image is then annotated to obtain training data for training the neural network.
  • Use these labeled data to train a neural network model (such as the YOLO network).
  • the recognition result of the neural network model includes a detection frame, which is used to represent the position of the optic disc in the fundus image.
  • a square frame 141 is used to mark the optic disc in the training data, and the recognition result of the trained neural network model will also be a square detection frame.
  • a circular frame may also be used for marking, or other similar marking methods are feasible.
  • the focal length can be continuously changed by means of gradient ascent and the corresponding fundus images can be collected to determine whether the clarity of the optic disc has reached the preset standard.
  • the best focal length does not need to continue searching; you can also use all available focal lengths within the adjustable focal length range, and collect the corresponding fundus images, determine a fundus image with the highest optic disc definition from all fundus images, and decide to collect the image.
  • the focal length at is the best focal length.
  • the traversal method is used to first adjust the focal length with a first set step size of 40 within the set focal length range of 800-1300, and collect the first group of fundus images, thus the fundus images when the focal length reaches 800 , the fundus image at the focal length of 840, the fundus image at the focal length of 880...the fundus image at the focal length of 1300.
  • the optic disc area is identified in these fundus images respectively, and the sharpness of each fundus image is determined respectively.
  • the mean value of the pixel values in the optic disc area is calculated as the sharpness.
  • a fundus image with the highest definition can be determined from the first group of fundus images, and at this time, the focal length X (first focal length) used when collecting the fundus image can be taken as the shooting focal length.
  • the focal length for example, perform another traversal near the above-mentioned focal length X, and the second set step size used in this traversal process is smaller than the above-mentioned first set step size, such as The second set step size is 10, so that the second group of fundus images can be obtained, namely, the fundus images at the focal length of X+10, the fundus images at the focal length of X+20, the fundus images at the time of X-10, and the fundus images at the time of X-20 fundus images, etc. Then, identify the optic disc area in these fundus images respectively, and determine the clarity of each fundus image. focal length) as the shooting focal length.
  • the first focal length X can be taken as the midpoint to increase the focal length of the first set step size to the maximum value and to decrease the focal length of the first set step size to the minimum value range, the range is X ⁇ 40.
  • fundus images are collected at different focal lengths, and whether the current focal length is suitable for capturing fundus images is judged by the clarity of the optic disc in the fundus images.
  • Hardware only need to set the image recognition algorithm to find the best focus position, which can reduce the cost of the fundus camera and improve the efficiency of focus adjustment.
  • This embodiment provides a preferred focal length adjustment method, which includes the following steps:
  • S1A use the current focal length to acquire a fundus image.
  • step S2A it is judged whether the subject blinks and/or closes his eyes according to the fundus image.
  • a prompt is given, for example, a voice prompts the user not to blink or close his eyes, etc., and then return to step S1A; otherwise, step S3A is performed.
  • Eye blinking and eye closing detection can also be realized by machine vision algorithm or neural network algorithm. When the subject blinks or closes his eyes, the collected image will be completely black or very blurry, and the features are relatively obvious. Various methods can be used for detection. , and will not be repeated here.
  • S3A identifying whether there is a light spot formed by the illumination beam reflected by the cornea in the fundus image.
  • the illumination beam reflected by the cornea should not be in the imaging range, and the above-mentioned light spot should not appear in the fundus image. , especially the complete imaging of the spot is not possible. Even if a light spot appears, it will be a part of the whole light spot.
  • a light source formed by a plurality of illuminating lamps arranged in a ring shape is used, and the complete light spot is shown in FIG. 12 .
  • a light spot appears in the fundus image when the focus is adjusted, it will be the situation as shown in FIG. 15 , in which only part of the light spot 151 is present. If the light source itself is a complete ring light, a band will appear in the image.
  • step S4A When there is a light spot in the fundus image, step S4A is performed, otherwise, step S5A is performed.
  • the vector offset can be calculated by combining the position, size and brightness of the light spot in the image.
  • a coordinate system is established with the image center as the origin (0,0), and the image radius is R.
  • the approximate circular area of each light spot 151 is calculated, and in this embodiment, the approximate circular area is the smallest circular area including the light spot 151 .
  • the center coordinate of the approximate circular area of the i-th light spot is (x i , y i ), and the radius is ri i .
  • step S1A After aligning the lens with the pupil again, the process returns to step S1A.
  • S5A Identify the optic disc area in the fundus image, and determine whether the clarity of the optic disc area reaches a set standard.
  • the mobilenet-yolov3 neural network model is used to identify the optic disc, and the optic disc area output by the neural network is the area including the optic disc and the background. Then, the edge of the optic disc is detected in the optic disc area by an edge detection algorithm (such as sobel, Laplace, etc.) to obtain an accurate optic disc image, and the mean value of the optic disc image is calculated as the sharpness value.
  • an edge detection algorithm such as sobel, Laplace, etc.
  • step S6A it can be determined whether the set standard is reached by comparing the obtained definition value with the threshold value. If the definition of the optic disc area does not meet the set standard, step S6A is executed. If the clarity of the optic disc area has reached the set standard, it is determined that the current focal length is suitable for capturing the fundus image, and then the infrared light can be turned off, and white light is used for exposure to capture the fundus image.
  • the initial focal length used in step S1A is the minimum value among the adjustable focal lengths, in this case, the focal length is increased according to a fixed step size or a variable step size, otherwise, the focal length is decreased.
  • the fundus image is started to be captured.
  • an illumination component needs to be used for exposure (the light source used by the camera in this embodiment is white light).
  • the subject may still affect the shooting quality of the fundus image, such as the pupil becomes smaller, the eyelid is blocked, the eye blinks, the light leakage of the face sticker component, etc., when these situations occur, the fundus image captured will appear unavailable Area.
  • this embodiment provides a fundus image shooting method, which can be executed by the fundus camera itself, or executed by an electronic device such as a computer or a server (as a control method), The method includes the following steps:
  • the lens is fixed at a position in the XY plane and aligned with the pupil, and positioned at the distance of the Z axis, using a fixed focal length, and the lens position, working distance and focal length remain unchanged. , expose the illumination assembly and capture multiple fundus images.
  • S2 respectively determine the quality of the multiple fundus images.
  • a neural network model is used to analyze image quality, and the neural network model can perform classification tasks to classify image quality, such as outputting classification results of high or poor quality; it can also perform regression prediction tasks to assess image quality. Perform quantification, such as outputting a score of 1-10 to express the evaluation of image quality.
  • a large number of retinal images exposed to white light are collected in advance, and the image quality is manually marked as good or bad (for classification models), or the image quality is scored (for example, 1 to 10 points, suitable for regression prediction models).
  • These fundus images and annotations or scores are used as training data to train a neural network model. After the model converges, it can be used to identify the quality of fundus images.
  • step S3 determine whether the quality of each fundus image meets the set standard, and if any fundus image meets the set standard, the fundus image can be used as the shooting result (the shooting result is output). If the quality of the multiple fundus images does not meet the set standard, step S4 is executed.
  • the lens state is kept unchanged, multiple fundus images are captured, and the quality of the multiple fundus images is determined respectively, and when it is determined that all the fundus images are unavailable, the multiple fundus images are used to synthesize A complete fundus image, even if the subject interferes with the shooting process, it will be possible to use the existing fundus image to obtain a high-quality fundus image, reduce the number of re-shots, reduce the user's difficulty in use, and improve the shooting of fundus. Image success rate.
  • an embodiment of the present invention provides a method for synthesizing a fundus image, the method comprising the following steps:
  • the brightness can be calculated according to the pixel value of the fundus image, and by comparing with the brightness threshold, the regions with higher brightness and the regions with lower brightness can be removed, thereby removing the overexposed and underexposed regions.
  • the area with moderate brightness that is, the high-quality area
  • the sharpness can also be calculated according to the pixel value of the fundus image, and the area with lower sharpness can be removed by comparing with the sharpness threshold, thereby removing the exposed blurred area, so as to obtain high-quality regions; or comprehensively extract high-quality regions based on brightness and sharpness.
  • the regions extracted according to the actual brightness and/or sharpness of the fundus image are usually regions with irregular boundaries, such as the two high-quality regions shown in Figure 16, the regions shown on the left are from the upper part of a fundus image, and the regions on the right The area shown is from the lower part of a fundus image.
  • each fundus image can also be divided into grids according to a fixed dividing method, and then the quality of each grid area is analyzed separately, and high-quality grids are extracted, so that high-quality areas with regular boundaries can be obtained. .
  • each fundus image may have some offset, in order to synthesize the fundus image more accurately, each fundus image can be mapped to the same coordinate system according to the offset, and then stitching and fusion processing are performed.
  • abnormal region detection is first performed on multiple fundus images to extract high-quality regions.
  • step S43 firstly, feature points (or key points) are extracted from the multiple fundus images respectively, which may be the center point of the optic disc, the intersection of blood vessels, or other significant positions. Then, feature point matching is performed to match the feature points between different fundus images. After these feature points are matched, the matching information is used to calculate the offset between the respective fundus images (projection matrix calculation). Then according to the offset, multiple high-quality regions are mapped into a fundus image.
  • the pixel values of the multiple high-quality regions and the corresponding weights can be used to determine the pixel values of the overlapping parts .
  • This is a fusion process based on weighted average.
  • the fusion process can be expressed as q1/(q1+q2)*image1+q2/(q1+q2)*image2, where q1 represents the first high-quality The weight of the region, q2 represents the weight corresponding to the second high-quality region, image1 represents the first high-quality region, and image2 represents the second high-quality region.
  • the value of the above weight is set according to the overall quality of the fundus image, for example, the first high-quality area is taken from the first fundus image, and the second high-quality area is taken from the second fundus image, and the value obtained according to the above quality analysis method.
  • the quality of the first fundus image (for example, the score output by the neural network) is higher than that of the second fundus image, so the corresponding weight q1 is greater than q2.
  • the fundus image synthesis method when there are defects in the multiple fundus images captured by the subject, high-quality regions are extracted from the multiple fundus images by using this scheme, and spliced and fused to obtain A complete fundus image with higher quality can be obtained, thereby reducing the difficulty for the user to take a self-portrait fundus image and improving the shooting success rate.
  • embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein, including but not limited to disk storage, CD-ROM, optical storage, and the like.
  • These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions
  • the apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

Landscapes

  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Biophysics (AREA)
  • Ophthalmology & Optometry (AREA)
  • Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Physics & Mathematics (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Eye Examination Apparatus (AREA)

Abstract

一种眼底相机及眼底图像全自动拍摄方法，眼底图像全自动拍摄方法包括：移动眼底相机镜头对准瞳孔(S300)；控制镜头接近眼球并采集图像(S400)，图像是对角膜所反射的照明光束的成像；利用图像确定工作距离(S500)；调整焦距并采集眼底图像，利用眼底图像确定拍摄焦距(S600)；在工作距离上使用拍摄焦距拍摄眼底图像(S700)。

Description

眼底相机及眼底图像全自动拍摄方法 技术领域
本发明涉及眼科仪器领域，具体涉及一种眼底相机及眼底图像全自动拍摄方法。
背景技术
视网膜是人体唯一可以直接观察到毛细血管和神经的组织，通过观察视网膜可以检查不仅仅眼部的健康问题还可以发现类似糖尿病并发症和高血压这样的全身的病变。眼底相机是用来拍摄视网膜的专用设备。
现有的眼底相机可以实现自动拍摄眼底图像，自动化拍摄的过程主要涉及自动将主镜头对准瞳孔、自动调整镜头和瞳孔的轴向距离(工作距离)，以及自动调整焦距。相机设有主摄像头、辅助摄像头以及众多辅助光学器件，主摄像头安装在一个可以X、Y、Z三个方向移动的平台上，用于拍摄眼底；辅助摄像头安装在主摄像头附近，用于拍摄脸部及外眼部，主要用于搜索眼部，实现自动对准瞳孔；辅助光学器件用于调焦、调整工作距离等等。
现有的眼底相机为了解决镜头对齐瞳孔，固定镜头和瞳孔的轴向距离，以及调焦问题需要复杂昂贵的硬件模块，使用起来也很复杂，阻碍了眼底相机的普及。
发明内容
有鉴于此，本发明提供一种眼底图像全自动拍摄方法，包括：
移动眼底相机镜头对准瞳孔；
控制所述镜头接近眼球并采集图像，所述图像是对角膜所反射的照明光束的成像；
利用所述图像确定工作距离；
调整焦距并采集眼底图像，利用所述眼底图像确定拍摄焦距；
在所述工作距离上使用所述拍摄焦距拍摄眼底图像。
可选地，在移动眼底相机镜头对准瞳孔之前，还包括：检测眼底相机的运动组件、照明组件和调焦组件是否正常。
可选地，检测眼底相机的运动组件、照明组件和调焦组件是否正常具体包括：
控制运动组件调节镜头的位置，检测所述镜头是否能够移动到各个定位组件所在的位置；
在所述镜头能够移动到各个定位组件所在的位置后，控制所述运动组件将所述镜头 移动到设定位置，开启照明组件并控制调焦组件调整为第一焦距，拍摄得到第一图像；
根据所述第一图像中的照明组件的图像特征判断所述调焦组件和所述照明组件是否正常；
当所述调焦组件和所述照明组件正常时，控制运动组件调节镜头至设定深度位置上，控制调焦组件调整为第二焦距，拍摄得到第二图像；
根据所述第二图像中的被拍物的图像特征判断成像功能是否正常。
可选地，在移动眼底相机镜头对准瞳孔之前，还包括：检测人体头部是否贴合眼底相机的面贴组件。
可选地，检测人体头部是否贴合眼底相机的面贴组件具体包括：
关闭照明组件，获取镜头透过面贴组件的视窗采集的第一图像；
判断所述第一图像的亮度是否达到设定标准；
当所述第一图像的亮度达到设定标准时，开启所述照明组件，获取镜头透过面贴组件的视窗采集的第二图像；
根据所述第二图像确定人体头部是否贴合所述面贴组件。
可选地，利用所述图像确定工作距离具体包括：
检测所述图像中的光斑的特征是否符合设定特征；
当所述光斑的特征符合设定特征时，确定达到工作距离。
可选地，利用所述眼底图像确定拍摄焦距具体包括：
在所述眼底图像中识别视盘区域；
根据所述视盘区域的清晰度确定拍摄焦距。
可选地，在所述工作距离上使用所述拍摄焦距拍摄眼底图像具体包括：
判断瞳孔尺寸是否小于眼底相机照明组件的环状照明光束尺寸；
当所述瞳孔尺寸小于所述环状照明光束尺寸时，分别向多个方向移动所述镜头与瞳孔产生偏移，使得所述环状照明光束部分照射在瞳孔中，并拍摄多个眼底图像；
将所述多个眼底图像融合为一个眼底图像。
可选地，在所述工作距离上使用所述拍摄焦距拍摄眼底图像具体包括：
获取镜头状态不变的情况下拍摄的多个眼底图像；
分别在所述多个眼底图像中提取高质量区域；
利用多个所述高质量区域合成眼底图像。
相应地，本发明提供一种电子设备，包括：至少一个处理器；以及与所述至少一个处理器通信连接的存储器；其中，所述存储器存储有可被所述一个处理器执行的指令， 所述指令被所述至少一个处理器执行，以使所述至少一个处理器执行上述眼底图像全自动拍摄方法。
相应地，本发明提供一种眼底相机，包括：面贴组件、运动组件、调焦组件、照明组件、镜头和至少一个处理器；以及与所述至少一个处理器通信连接的存储器；其中，所述存储器存储有可被所述一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器执行上述眼底图像全自动拍摄方法。
根据本发明提供的眼底相机及眼底图像全自动拍摄方法，能够使眼底相机自动将主镜头对准瞳孔、自动调整工作距离，以及自动调整焦距，本方案通过图像识别算法实现眼底相机全自动拍摄眼底图像，不需要辅助摄像头和辅助光学器件，降低了硬件的复杂度和使用难度，让用户能够自主拍摄眼底图像，促进眼底相机的普及。
附图说明
为了更清楚地说明本发明具体实施方式或现有技术中的技术方案，下面将对具体实施方式或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图是本发明的一些实施方式，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本发明实施例中的眼底相机的结构图；
图2为本发明实施例中的眼底相机的面贴组件的示意图；
图3为镜头及定位组件的示意图；
图4为本发明实施例中的一种眼底图像全自动拍摄方法的流程图；
图5为瞳孔标注示意图；
图6为本发明实施例中优选的眼底图像全自动拍摄方法的流程图；
图7为瞳孔大于照明光束的示意图；
图8为瞳孔小于照明光束的示意图；
图9为瞳孔小于照明光束时拍摄眼底图像的示意图；
图10为角膜反射照明光束的成像；
图11为镜筒与眼球间的距离示意图；
图12为光斑标注示意图；
图13为达到工作距离时角膜反射照明光束的成像；
图14为视盘标注示意图；
图15为拍摄眼底图像时根据光斑移动镜头位置的示意图；
图16为两个存在不可用区域的眼底图像的示意图；
图17为眼底图像的合成方式示意图；
图18为一种照明灯的结构图；
图19为检测相机状态时对照明反射光的成像示意图；
图20为检测相机状态时对面贴组件凸起部的成像示意图；
图21为检测相机状态时对设有标靶的面贴组件凸起部的成像示意图；
图22为检测被拍者使用状态时采集的双眼之间区域的图像。
具体实施方式
下面将结合附图对本发明的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
在本发明的描述中，需要说明的是，术语“中心”、“上”、“下”、“左”、“右”、“竖直”、“水平”、“内”、“外”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本发明和简化描述，而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作，因此不能理解为对本发明的限制。此外，术语“第一”、“第二”、“第三”仅用于描述目的，而不能理解为指示或暗示相对重要性。
在本发明的描述中，需要说明的是，除非另有明确的规定和限定，术语“安装”、“相连”、“连接”应做广义理解，例如，可以是固定连接，也可以是可拆卸连接，或一体地连接；可以是机械连接，也可以是电连接；可以是直接相连，也可以通过中间媒介间接相连，还可以是两个元件内部的连通，可以是无线连接，也可以是有线连接。对于本领域的普通技术人员而言，可以具体情况理解上述术语在本发明中的具体含义。
此外，下面所描述的本发明不同实施方式中所涉及的技术特征只要彼此之间未构成冲突就可以相互结合。
图1示出了一种全自动便携自拍眼底相机，该相机包括面贴组件01、运动组件、定位组件03和镜筒1，镜筒1内部设有照明组件、调焦组件、镜头(接目物镜)以及光学镜片组和成像探测器10等，镜筒1的内部结构可参考中国专利文件CN111134616A。实际产品还包括壳体，运动组件和镜筒1位于壳体内部。面贴组件01密封地连接于壳体的前部，面贴组件包括面贴本体和成型于面贴本体的用于被拍摄者的眼部贴合时容纳所述眼部的视窗通孔。面贴组件01作为接触被拍摄者眼部的部件，镜筒1透过面贴组件01的通孔采集被拍摄者的眼底视网膜图像。
面贴本体背向镜筒1的一面被构造为与被拍摄者的眼部周围的面部轮廓相贴合的形状。具体地，面贴组件01向内形成凹陷形状以适于人体头部弧形，其通孔的尺寸至少能在被测者的眼部贴合本组件时容纳双眼。面贴组件01向内(壳体内、镜筒)的一面有至少一个用于检测相机各项功能的特定位置。在一个具体的实施例中，结合图1和图2所示，图2所展示的是面贴组件01朝内的一面，通孔011的中部上边缘有一个凸起部012，镜筒1的镜头能够对准此部位并拍摄图像。更优选的方案是，在此凸起部012上设置一个图案或者简单图形等作为标靶。此特定位置有多种用途，包括检测相机的照明组件、调焦组件是否正常，检测被拍者的眼部是否正确地贴合了面贴组件01等，具体将在下文中进行详细介绍。
运动组件用于控制镜筒1在三维空间中移动，以图1中的坐标系为例，能够在图中的X、Y、Z三轴上移动。需要说明的是，镜筒1在Z方向移动到极限位置时，端部不会伸出面贴组件01外。作为一个具体的实施例，运动组件包括三个轨道组件，第一组轨道021用于控制镜筒1在X轴上的运动、第二组轨道022用于控制镜筒1在Y轴上的运动、图中未示出的第三组轨道用于控制镜筒1在Z轴上的运动。具体地，镜筒1连同第二组轨道022被设置在一个平台(基座)上，第一组轨道021能够带动基座整体运动，第三组轨道能够带动基座和第一组轨道021运动，使整体接近或远离面贴组件01。
定位组件03用于检测镜筒1的移动情况。具体地，定位组件03可以是电磁感应器，根据电磁感应信号感知镜筒1移动到定位组件03所在的位置。结合图3所示，在本实施例中设置3个定位组件03，其中两个定位组件设置在可移动基座两侧，以检测镜筒1在X轴上的移动，第三个定位组件设置在该基座上，以检测镜筒1在Y轴上的移动，即定位组件03用于检测镜筒1在XY平面内的移动情况。
根据本发明提供的眼底相机，用于成像的照明组件、调焦组件、接目物镜、光学镜片组和成像探测器集成在一个镜筒中实现光路结构小型化，减小眼底相机的体积，提高便携性；眼底相机的面贴组件设有用于容纳被拍摄者眼部的视窗通孔，用户可以自行佩戴眼底相机，将眼部置于视窗通孔位置，运动组件驱动镜筒在视窗通孔范围中搜索瞳孔，并调整工作距离，从而拍摄眼底图像，本方案降低了眼底相机硬件的复杂度和使用难度，让用户能够自主拍摄眼底图像，促进眼底相机的普及。
本发明实施例提供一种眼底图像全自动拍摄方法，该方法可以由眼底相机本身来执行，也可以由计算机或者服务器等电子设备执行(作为一种控制方法)。如图4所示，该方法包括如下步骤：
S300，移动眼底相机镜头对准瞳孔。
S400，控制镜头接近眼球并采集图像，该图像是对角膜所反射的照明光束的成像。
S500，利用上述图像确定工作距离。
S600，调整焦距并采集眼底图像，利用眼底图像确定拍摄焦距。
S700，在工作距离上使用拍摄焦距拍摄眼底图像。
在优选的实施例中，在上述步骤S100之前还可以执行检测相机状态和用户使用状态的步骤，如图6所示,本方法还可以包括：
S100，检测眼底相机的运动组件、照明组件和调焦组件是否正常。此步骤作为可选操作，可以在眼底相机开机时执行。如果检测到某部件异常将终止后续拍摄操作并进行相应的异常提示。
S200，检测人体头部是否贴合眼底相机的面贴组件。此步骤作为可选操作，如果检测到人体头部未贴合眼底相机的面贴组件，可以通过语音模块向用户进行提示，引导用户正确佩戴眼底相机。
针对上述步骤S100，本发明实施例提供一种眼底相机检测方法，该方法可以由眼底相机本身来执行，作为一种自检方法，也可以由计算机或者服务器等电子设备执行，作为一种产品检测方法，该方法包括如下步骤：
S1，控制运动组件调节镜头的位置，检测镜头是否能够移动到各个定位组件所在的位置。本方法适于在眼底相机刚刚被启动时执行，首先镜头(根据上述实施例，镜头与镜筒是一体设置，移动镜筒即为移动镜头)被移动到初始位置上。然后结合图3所示，运动组件调节镜头的位置，检测镜头是否能够移动到这3个定位组件所在的位置。如果能够移动到这些位置则认为运动组件功能正常，可以执行步骤S2，否则执行步骤S6。步骤S1可称为运动组件XY轴运动检测步骤。
S2，控制运动组件将镜头移动到设定位置，开启照明组件并控制调焦组件调整为第一焦距，拍摄得到第一图像。此步骤的目的是检测调焦组件和照明组件的功能是否正常，理论上不需要限定镜头必须对准某个位置，所以此步骤中所述的设定位置有多种选择。但在实际工作环境中外部环境是不确定的，比如可能是一个比较明亮的环境，如果此步骤拍摄第一图像时外部环境比较明亮，可能导致图像内容受到干扰。为了适用于实际工作环境，此步骤将镜头移动到对准面贴组件的某个特定部位(比如上述凸起部)，尽可能少地拍摄到外部环境，图像中面贴组件的占比大于外部环境的占比。当然，修改面贴组件及其通孔的形状，使此步骤所拍摄的图像中完全不包含外部环境也是可能的。
通过设置合适的焦距，可以使照明组件成像。比如图18示出了一个镜筒中的照明灯的结构，一个环形结构上面设有4个灯珠。开启这4个灯珠，使用第一焦距进行成像， 预期得到一个如图19所示的图像。
在优选的实施例中，为了避免拍摄的图像中背景对照明组件成像的影响，所设置的第一焦距能够使得照明组件成像而不能使面贴组件成像。由此使得所述第一图像中只可能存在照明组件而不存在如面贴组件凸起部等物体，提高后续步骤中对图像识别的准确性。
S3，根据第一图像中的照明组件的图像特征判断调焦组件和照明组件是否正常。在调焦组件功能正常的情况下，使用设定的焦距应当能够得到图19所示的图像，此图像具有明显的特征，具体取决于照明组件的实际形状。比如在本实施例中，第一图像中应当有4个独立且清晰的点状物，这是对上述4个灯珠的成像结果。如果此时调整的焦距不是上述第一焦距，则图像中的点状物将会变大且模糊，或者变得更小；如果照明组件没有开启，则图像中不会出现形状。
通过机器视觉算法或者神经网络算法，对第一图像进行识别，都可以识别出图像中是否存在符合预期的特征。如果判定调焦组件和照明组件正常，则执行步骤S4，否则执行步骤S6。步骤S2-S3可称为调焦组件和照明组件检测步骤。
S4，控制运动组件调节镜头至设定深度位置上，控制调焦组件调整为第二焦距，拍摄得到第二图像。在此步骤中，需要对一个已知物体进行成像，作为优选的实施例，将面贴组件的凸起部作为已知物体。具体地，首先将镜头在XY平面上对准面贴组件的凸起部，在本实施中步骤S2已经对准此部位，此步骤不需要再做调整；在其它实施例中，如果步骤S2中未对准此部位，则在此步骤进行调整。此步骤需要调整深度，也即在Z轴上调整镜头的位置，可以理解为调整对已知物体的拍摄距离，然后再设定焦距。
为了能够使外部物体成像，此时的焦距与步骤S2中所使用的焦距是不同的，并且此步骤中的焦距应当适用于当前的镜头位置(深度位置)，预期得到一个如图20所示的图像。
在优选的实施例中，为了避免拍摄的图像中出现照明组件的影像对被拍物体成像的影响，所设置的第二焦距能够使得面贴组件成像而不能使照明组件成像。由此使得所述第二图像中只可能存在被拍物如面贴组件凸起部，而不会出现照明组件的影像，提高后续步骤中对图像识别的准确性。
S5，根据第二图像中的被拍物的图像特征判断成像功能是否正常。第二图像是运动组件在XY轴的运动、照明组件和调焦组件均正常的情况下拍摄的图像，此步骤的目的是检测运动组件在Z轴的运动是否正常，如果步骤S4中运动组件能够将镜头调整至设定深度位置上，拍摄的第二图像中应当能够展现出清晰的被拍物，比如图20所示的面 贴组件的凸起部。
通过机器视觉算法或者神经网络算法，对第二图像进行识别，都可以识别出图像中是否存在符合预期的特征。如果判定运动组件在Z轴的运动正常，则结束检测，判定眼底相机各主要组件功能正常，否则执行步骤S6。步骤S4-S5可称为运动组件Z轴运动检测步骤。
S6，判定眼底相机状态异常。根据出现异常的组件，向用户进行具体故障部位提示。眼底相机中可设置语音模块或者信息显示模块，向用户播报或者显示相应的故障信息。
根据本发明实施例提供的眼底相机检测方法，通过定位组件来验证运动组件是否可以正常调整镜头的位置；在确认运动组件正常后，调整焦距使照明组件成像，通过对采集的图像进行判断可以确定调焦组件和照明组件是否正常；最后通过运动组件调整镜头的深度，调整焦距使被拍物成像，对图像中的物体特征进行判断，来验证运动组件是否可以正常调整镜头的深度，由此自动确定眼底相机各个重要部件是否能够正常工作。本方案可以在远程无人值守环境下进行自助检查设备工作状态，由此可以提高拍摄眼底照片的便利性，促进眼底相机的普及。
在优选的实施例中，面贴组件的凸起部设有标靶，即上述被拍物为面贴组件的设定部位上的标靶，标靶的具体内容不限，是一个或者多个轮廓清晰的图案或形状都是可行的。所得到的第二图像如图21所示，其中包括一个圆形标靶81，步骤S5具体包括：
S51，识别第二图像中是否存在清晰的标靶的影像；
S52，当第二图像中存在清晰的标靶的影像时确定成像功能正常。
利用机器视觉算法或者神经网络算法识别标靶的结果更加准确，如果图像中不存在标靶轮或者轮廓不清楚，将更容易地被识别到，从而进一步提高对相机功能判断的准确性。
针对上述步骤S200，本发明实施例提供一种眼底相机使用状态检测方法，用于检测用户是否正确地佩戴了上述实施例中的眼底相机。该方法可以由眼底相机本身来执行，作为一种自检方法，也可以由计算机或者服务器等电子设备执行。本方法适于在根据上述检测方法确定相机各重要部件正常之后执行，该方法包括如下步骤：
S1，获取镜头透过面贴组件的视窗采集的第一图像。在本方案中，镜头透过如图2所示的通孔011采集外部环境的图像，应当避免面贴组件遮挡镜头(面贴组件不在成像范围内)，当被拍者正确佩戴了眼底相机时，眼部与面贴组件01贴合，视窗(通孔011)中是人体双眼及周围皮肤，镜头采集到相应的第一图像。在此步骤中需要保持照明组件处于关闭状态，也即没有通过镜头向外照射光束。在本方案中，由于对采集图像的清晰 度要求不高，采集图像所使用的焦距可以是一个固定值，将成像平面大致设在人体表面即可。当然也可以先开启照明组件，进行自动对焦，将成像平面更精确地设在人体表面后再关闭照明组件。
S2，判断第一图像的亮度是否达到设定标准。如果被拍者的眼部与面贴组件01贴合，四周没有大的缝隙，采集的第一图像应当是很暗的。在此先对第一图像的亮度进行判断，如果亮度达到设定标准则执行步骤S3，否则执行步骤S6。
判断图像的亮度是否符合设定标准的方法有多种，比如可以根据图像的像素值计算亮度值，然后与阈值进行比较；也可以采用神经网络算法，预先使用不同亮度的图像对神经网络进行训练，使其具备对图像亮度的分类或者回归预测能力，在此使用神经网络对第一图像进行识别，输出关于亮度的识别结果。
在一个优选的实施例中，将第一图像转换为灰度图像，然后再识别灰度图像的亮度。
S3，开启照明组件，获取镜头透过面贴组件的视窗采集的第二图像。此时镜头和被拍者的状态未发生变化，只是开启照明光源，通过镜头向外进行照明，此时照明光束照射到被拍者的眼部或皮肤并被反射。在一个优选的实施例中，将镜头的位置设为对准面贴组件的视窗的中心位置，所使用的光源是红外光，如果人体头部贴合面贴组件，则镜头对准的是双眼之间的区域，可采集到如图22所示的图像。
S4，根据第二图像确定人体头部是否贴合面贴组件。如果被拍者的头部与面贴组件贴合，由于人体皮肤会反射照明光束，因此如图22所示的图像中会出现一个明显的光斑，并且光斑四周会呈现人体皮肤特征，通过判断图像是否具有中心较亮、边缘渐暗的特征，即可确定人体头部是否贴合面贴组件。
假设在步骤S1-S2中没有物体贴合面贴组件，相机置于一个暗室中，或者面贴组件被其它物体遮盖，第一图像的亮度也将被判定为符合设定标准，这需要进一步执行步骤S3-S4进行判断。如果没有物体贴合面贴组件，采集的第二图像中不会出现光斑；如果是其它物体遮盖了面贴组件，第二图像中会出现光斑，但由于材质、表面形状不同，对于照明光束的反射情况与人体不同，因此通过光斑的特征即可判断出是否为人体。
在其它可选实施例中，采集第一、第二图像时镜头也可以对准其它位置，比如对准眼球，在步骤S4中可以在图像中识别眼球特征来确定是否为人体。
在优选的实施例中，步骤S4可以首先判断第二图像的亮度是否达到设定标准，与识别第一图像的亮度类似地，可以将第二图像转换为灰度图像，再计算亮度值，或者利用神经网络进行识别。如果此时面贴组件与人体贴合处出现缝隙而出现漏光情况，受到环境光的影响，第二图像的亮度与只有相机本身的光源进行照明时的亮度不同。在排除 漏光情况后，再判断第二图像中的特征是否符合人体皮肤特征。
当确定人体头部贴合面贴组件时执行步骤S5，否则执行步骤S6。
S5，开始拍摄眼底图像。具体地，需要自动寻找瞳孔、调整工作距离、调整焦距将成像平面设在眼底，最终拍摄得到眼底图像。
S6，提示用户正确佩戴眼底相机。比如可以在眼底相机中设置语音模块，提示用户如何正确佩戴眼底相机等等，之后可以返回步骤S1重新判断。
根据本发明实施例提供的眼底相机使用状态检测方法，在关闭照明组件的情况下采集图像，通过图像的亮度可以初步判断面贴组件是否被物体良好遮盖，然后在开启照明组件的情况下采集图像，通过图像特征进一步判断遮盖物是否为人体，由此自动确定被拍者是否正确地佩戴了眼底相机、是否在合适的环境中使用眼底相机，本方案可以全自动触发眼底相机拍摄眼底照片，不需要手动干预触发拍摄，不需要专业人员进行操控，由此可以提高拍摄眼底照片的便利性，促进眼底相机的普及。
在相机开始拍摄的时候，实际应用场景下的瞳孔和接目物镜是不会完全对齐的，这时候需要相机通过瞳孔在传感器的成像来判断镜头相对瞳孔的位置，然后把镜头移动到瞳孔的正前方，再进行拍摄。针对上述步骤S300，本发明实施例提供一种眼底相机镜头自动对准方法，该方法可以由眼底相机本身来执行，也可以由计算机或者服务器等电子设备执行(作为一种控制方法)，该方法包括如下步骤：
S1,对眼底相机镜头采集的图像进行识别，判断其中是否存在瞳孔。具体地，当用户佩戴上述眼底相机后，系统会连续(比如逐帧地)采集瞳孔的图像，如果能够在图像中识别到瞳孔，说明瞳孔已经在成像范围内，在此情况下进行微调以使镜头完全对准瞳孔即可进行拍摄。如果不能在图像中识别到瞳孔，则说明镜头与瞳孔位置偏差较大，原因可能是镜头的初始位置不合适，或者用户佩戴方式不标准等等。
在图像中识别瞳孔影像的方式有多种，例如可以使用机器视觉算法，根据图像中的图形特征检测瞳孔轮廓和位置。然而，由于眼底相机在最终拍摄之前是使用红外光进行照明，因此瞳孔的成像不会非常清晰，同时角膜的反光也很给瞳孔检测带来很多困难，计算机视觉算法遇到这种情况很容易误判，因此在一个优选的实施例中使用深度学习算法来解决这个问题。
首先采集大量瞳孔的照片，这些照片是不同人、在距离上述眼底相机的接目物镜不同方向和距离、不同时间采集到的图像。然后在对每一张图像中的瞳孔进行标注，由此得到用于训练神经网络的训练数据。使用这些标注好的数据来训练一个神经网络模型(比如YOLO网络)，经过训练后，神经网络模型的识别结果包括一个检测框，用于表征 图像中瞳孔的位置和尺寸。
如图5所示，在一个具体的实施例中，训练数据中采用方形框51来标注瞳孔，训练后的神经网络模型的识别结果也将是方形检测框。在其它实施例中，也可以使用圆形框进行标注，或者其它类似的标注方式都是可行的。
无论采用何种瞳孔检测方法，在此步骤中只需要识别图像中是否存在瞳孔即可，如果图像中不存在瞳孔则执行步骤S2，否则执行步骤S3。
S2，控制眼底相机镜头在当前位置附近进行移动以搜索瞳孔。通过上述运动组件移动镜筒，例如做螺旋状轨迹移动，从当前位置开始逐渐向周围扩散。需要说明的是，本实施例只涉及到上述XY平面内的移动，暂时不论述Z轴的移动，Z轴的移动关系到眼底相机的最佳工作距离，具体将在后续的实施例中进行介绍。
如果移动到极限位置后仍不能搜索到瞳孔，则提示用户调整佩戴状态；如果搜索到瞳孔，进一步判断如果用户的眼部是否偏离镜头很远，超出了运动组件能够移动的范围，比如判断镜头的移动距离是否超过移动阈值，当移动距离超过移动阈值时提示用户在面贴组件内轻微移动头部，以适应镜头的移动范围。然后继续搜索，当移动距离未超过移动阈值时执行步骤S3。
S3，判断图像中的瞳孔是否符合设定条件。具体可设置多种设定条件，比如关于尺寸的条件、关于形状的条件等。
在一个可选的实施例中，设定条件包括尺寸阈值，判断图像中的瞳孔尺寸是否大于尺寸阈值，当图像中的瞳孔尺寸大于尺寸阈值时，判定存在符合设定条件的瞳孔；否则提示用户闭眼休息一段时间，使瞳孔放大之后再开始拍摄。因为拍摄眼底图像时一般是需要依次对双眼进行拍摄，对第一只眼拍摄之后会导致瞳孔缩小，因此系统也会让用户闭眼休息，让瞳孔大小恢复。
在另一个可选的实施例中，设定条件包括形态特征，判断图像中的瞳孔形状是否符合设定的形态特征，当图像中的瞳孔形状符合设定的形态特征时，判定存在符合设定条件的瞳孔；否则提示用户睁大眼睛、尽量不要眨眼等等。设定的形态特征为圆形或者近似圆形，如果检测出来的瞳孔不符合预设的形态特征，比如可能是扁平的，这种情况一般是因为用户眼睛没有睁开所导致的。
在第三个可选的实施例中，需要使用上述神经网络模型进行瞳孔检测，并且神经网络模型的识别结果还包括瞳孔的置信度信息，也即用于表示模型判定图像中存在瞳孔的概率值。所述设定条件包括置信度阈值，判断神经网络模型得到的置信度信息是否大于所述置信度阈值。当所述置信度信息大于置信度阈值时，判定存在符合设定条件的瞳孔； 否则提示用户睁大眼睛，移除头发等遮挡物。神经网络模型得到的瞳孔的置信度比较低，说明图像中虽然存在瞳孔但其可能被其它物体所干扰，为了提高拍摄质量，在此提示用户进行调整。
上述三种实施例可以被择一使用，也可以组合使用。当图像中的瞳孔符合设定条件时执行步骤S4，否则等待用户的调整自身状态并持续判断直至符合设定条件。
S4，根据瞳孔在图像中的位置移动眼底相机镜头对准瞳孔。通过上述运动组件移动镜筒，移动方向和距离取决于图像中的瞳孔与镜头的偏差。将采集的图像中心点视为镜头的中心点，并识别图像中瞳孔的中心点。关于识别图像中瞳孔的中心点的方式，比如在使用上述神经网络模型检测瞳孔时，将检测框的中心点视为瞳孔的中心点即可。步骤S4具体包括：
S41，根据检测框的中心位置和所述图像的中心位置的偏差确定移动距离和移动方向；
S42，根据确定的移动距离和移动方向移动眼底相机镜头对准瞳孔。
根据本发明实施例提供眼底图像拍摄方法，通过对图像中的瞳孔状态进行判断，可以自动确定被拍者当前的瞳孔状态是否适合拍摄眼底图像，在其状态不适于拍摄眼底图像时，可以向被拍者发出相应的提示以使其调整自身的状态，在其状态适合拍摄眼底图像时，识别瞳孔的位置从而进行自动对准，之后进行拍摄，由此避免拍摄到不可用的眼底图像，整个过程不需要专业人员参与，实现用户自主拍摄。
在实际应用场景中，可能出现一种特别的情况，即瞳孔的尺寸可能小于环状照明光束的尺寸，在此情况下将瞳孔和接目物镜对齐会导致没有任何光线能进入瞳孔，因此拍摄出来的图像是黑的。
为了解决这一问题，针对上述步骤S700，本发明实施例提供一种优选的眼底图像拍摄方法，本方法包括如下步骤：
S51，判断图像中的瞳孔尺寸是否小于眼底相机照明组件的环状照明光束尺寸。图7示出了一个瞳孔72尺寸大于环状光束71尺寸的情况，此情况下执行步骤S52。
图8示出了两个环状照明光束尺寸大于瞳孔尺寸的情况，照明光源是一个完整的环形照明灯或者是由多个照明灯呈环状排列形成的光源，环状光束71的内径大于瞳孔72的直径。
当瞳孔尺寸小于环状照明光束尺寸时，即符合如图8所示情形时执行步骤S53。
S52，以当前的镜头位置拍摄眼底图像。这是在光源良好地照射眼底的情况下所拍摄的图像。
S53，分别向多个方向移动镜头与瞳孔产生偏移，使得环状照明光束部分照射在瞳孔中，并获取多个眼底图像。以图9所示移动为例，在本实施例中，分别向水平的两个方向移动镜头，当镜头向一侧移动后使得环状光束71的一部分73照射到瞳孔72中，此时拍摄一张眼底图像；当镜头向另一侧移动后使得环状光束71的另一部分74照射到瞳孔72中，此时拍摄另一张眼底图像。
图9所示的移动和照明只是为了说明拍摄情况而做的举例，实际应用中可以向更多方向移动以拍摄更多的眼底图像。然而类似这种移动和照明情况下所拍摄的眼底图像会有部分区域出现过曝现象，这种眼底图像无法直接作为拍摄结果，因此执行步骤S54。
另外，为了减少过曝光的区域，在一个优选的实施例中，采用如下方式移动和拍摄：
S531，确定瞳孔的边缘位置。具体可使用机器视觉算法或者使用上述神经网络模型，得到图9中瞳孔72的左侧边缘点721和右侧边缘点722。
S532，根据瞳孔的边缘位置确定移动距离。具体可根据当前镜头中心的位置O(图像中心位置)与左侧边缘点721和右侧边缘点722的位置关系计算运动组件的移动距离。
S533，按照确定的移动距离分别向多个方向移动所述镜头，所确定的移动距离使得环状照明光束的边缘与瞳孔的边缘位置重合。如图9所示，环状光束71的外圈边缘恰好与瞳孔72的边缘重合，这样可以使环状光束71进入眼底的部分位于眼底的边缘，减少对眼底中心区域成像的影响。
S54，将多个眼底图像融合为一个眼底图像。在此步骤中，将分别从各个眼底图像中提取可用的区域，利用这些眼底图像拼接并融合出一张完整的眼底图像。拼接和融合的方式有多种，作为一个可选的实施例，步骤S54具体包括：
S541a，根据获取的眼底图像对应的镜头移动距离计算多个眼底图像的位移偏差；
S542a，在多个眼底图像中选取有效区域；
S543a，根据位移偏差对多个有效区域进行拼接，得到拼接后的眼底图像。进一步地，利用图像融合算法在各个所述有效区域的拼接处进行融合处理。
作为另一个可选的实施例，步骤S54具体包括：
S541b，在多个眼底图像中检测相应的特征点；
S542b，根据特征点的位置计算多个眼底图像的空间变换关系；
S543b，根据空间变换关系将多个眼底图像设置在同一坐标系下；
S544b，在处于同一坐标系下的多个眼底图像中选取有效区域进行拼接，得到拼接后的眼底图像。
根据本发明实施例提供眼底图像拍摄方法，在眼底相机镜头对准瞳孔时，首先判断 比较图像中的瞳孔尺寸和相机本身发出的环状光速的尺寸，如果瞳孔尺寸太小导致照明光束不能正常照射到眼底，则移动镜头，偏离当前的对准位置使得环状照明光束部分照射在瞳孔中，并在多个偏移位置上获取眼底图像，最终根据多个眼底图像融合出一个眼底图像，本方案可以在被拍者瞳孔较小的情况下拍摄眼底图像，不需要专业人员参与拍摄过程，降低对被拍摄者瞳孔状态的要求，提高拍摄效率。
下面介绍关于相机镜头(镜筒)在Z轴的移动，Z轴的移动关系到眼底相机的最佳工作距离。针对上述步骤S400-S500，本实施例提供一种眼底相机的工作距离调整方法，该方法可以由眼底相机本身来执行，也可以由计算机或者服务器等电子设备执行(作为一种控制方法)。该方法包括如下步骤：
S1，控制镜头接近眼球并采集图像，该图像是对角膜所反射的照明光束的成像。此步骤是根据上述实施例的方案，在XY平面内将镜头对准瞳孔的情况下被执行，此步骤中控制镜头接近眼球是指通过运动组件控制镜头在Z轴上向眼球的方向移动。在初始的距离上，照明组件的光源通过光学镜头，照射到眼睛角膜上的反射光在cmos上进行成像会得到如图10所示的结果，在本实施例中的光源是按十字形分布于照明组件四侧的四个灯球，对此光源的成像中也相应地显示出四个光斑。在其它实施例中，照明光源可以是如图8所示的形状，所采集的图像中将显示出相应形状或排列的光斑。
S2，检测图像中的光斑的特征是否符合设定特征。如图11所示，随着镜筒1在Z轴上向眼球01移动，角膜反射光成像将会发生变化。具体来说，成像的位置、尺寸和清晰度，与接目物镜和角膜之间的距离相关。距离越近，入射光线和角膜的法线夹角越大，反射的散射效果更重，光斑尺寸越大、越发散、亮度越低。
在图像中识别光斑特征的方式有多种，例如可以使用机器视觉算法，根据图像中的图形特征检测光斑轮廓和位置。然而，由于光斑的清晰度、尺寸等各方面的变化范围较大，计算机视觉算法遇到这种情况很容易误判，因此在一个优选的实施例中使用深度学习算法来解决这个问题。
首先采集大量光斑的成像，这些成像是不同人、在距离上述眼底相机的接目物镜不同方向和距离、不同时间采集到的图像。然后在对每一张图像中的光斑进行标注，由此得到用于训练神经网络的训练数据。使用这些标注好的数据来训练一个神经网络模型(比如YOLO网络)，经过训练后，神经网络模型的识别结果包括一个检测框，用于表征图像中光斑的位置和尺寸。
如图12所示，在一个具体的实施例中，训练数据中采用方形框121来标注光斑，训练后的神经网络模型的识别结果也将是方形检测框。在其它实施例中，也可以使用圆 形框进行标注，或者其它类似的标注方式都是可行的。
无论采用何种光斑检测方法，在此步骤中识别到当前图像中的光斑特征符合设定特征即可。所述设定特征可以是关于尺寸的特征，比如当图像中的光斑尺寸小于设定尺寸时判定符合设定特征；也可以是光斑消失，比如当使用机器视觉算法或者神经网络不能检测到图像中的光斑时判定符合设定特征。
如果图像中的光斑符合设定特征则执行步骤S3，否则返回步骤S1，继续移动镜头并采集图像。
S3，确定达到工作距离。在确定图像中光斑的特征符合设定特征时，此时镜头与眼球之间的距离可以被视为达到工作距离。在具体的实施例中，根据硬件参数，还可以在此距离的基础上再进行一个距离补偿，补偿的方向、距离值与硬件参数有关。作为举例，图13示出了一个光斑符合设定特征的图像，此时镜头1与眼球01的距离为WD，在此基础上控制镜头继续向接近眼球的方向移动预设距离d，以达到更准确的工作距离WD+。
在此工作距离上，进一步调整焦距即可拍摄眼底图像。关于调整焦距的方式，具体将在后续的实施例中进行介绍。
根据本发明实施例提供的工作距离调整方法，对角膜所反射的照明光束的成像进行采集和识别，通过图像中的光斑特征来判断和调整镜头与眼球的距离，不需要在眼底相机上设置任何额外的光学或者硬件，只需要设置合适的照明光束即可实现准确定位工作距离，由此可以降低眼底相机的成本，并提高工作距离调整效率。
考虑到镜头向眼球方向移动的过程中，用户可能会轻微转动头部等等，这将导致镜头不再是对准瞳孔的状态，因此在调整工作距离的过程中，还将在XY平面上调整镜头的位置以保持对准瞳孔。本实施例提供一种优选的调整工作距离的方法，该方法包括如下步骤：
S1A，采集角膜所反射的照明光束的成像；
S2A，调用神经网络对图像中的光斑进行检测，判断图像中是否存在光斑。当图像中不存在光斑时执行步骤S6A，否则执行步骤S3A。
S3A，识别图像中光斑的中心点，并判断光斑的中心点与图像的中心点是否重合。在此将神经网络得到的检测框的中心视为光斑的中心。图像的中心点被视为镜头的中心，如果图像的中心点与光斑中心重合则表示镜头对准了瞳孔，执行步骤S5A；二者不重合则表示镜头偏离的对准位置，执行步骤S4A。
S4A，根据光斑的中心点与图像的中心点的偏移量微调镜头位置。检测-调整-再检测是一个反馈过程，作为一个优选的实施例，在此使用一种平滑的调整算法：
Adjustment(i)＝a*Shift(i)+(1-a)Adjustment(i-1)，
其中Adjustment(i-1)表示上一次镜头调整的位移，Shift(i)表示所述偏移量(瞳孔中心和图像中心的偏差)，Adjustment(i)表示本次镜头需要调整的位移，a为0-1之间的系数。因为镜头的位置是XY平面上的二维坐标，Adjustment和Shift都是二维向量。
将光斑的中心点与图像的中心点调整至重合后，执行步骤S5A。
S5A，控制镜头向接近眼球的方向移动，以缩小距离。之后返回步骤S1A，随着反复执行使镜头逐渐接近眼球，相应的图像中的光斑尺寸由大变小。为了精确地捕捉到光斑消失的临界点，可以对每一帧图像进行采集并相应地做出上述判断和调整，直到检测到光斑消失的图像为止。
S6A,控制镜头继续向接近眼球的方向移动预设距离，以达到工作距离。
在优选的实施例中，在执行上述调整过程的同时，还将检测图像中的光斑是否完整，当光斑不完整时，比如只有一半，这意味着用户在眨眼或者眼睛没有睁开，此时系统会通过语音提示用户睁大眼，尽量不要眨眼等等。
根据本发明实施例提供的工作距离调整方法，在调整镜头与眼球的距离的同时，还将根据图像中的光斑位置微调镜头位置，由此在调整工作距离时保持镜头对准瞳孔，本方案不需要在眼底相机上设置任何额外的光学或者硬件，只需要设置合适的照明光束即可实现准确定位工作距离并保持镜头对准瞳孔，由此可以降低眼底相机的成本，并提高眼底图像拍摄效率。
通过上述实施例的自动对准和自动调整工作距离之后，还需要设置合适的焦距才能拍摄到清晰的眼底图像。下面介绍关于自动调整焦距的技术方案，针对上述步骤S600，本实施例提供一种眼底相机的焦距调整方法，该方法可以由眼底相机本身来执行，也可以由计算机或者服务器等电子设备执行(作为一种控制方法)。该方法包括如下步骤：
S1，调整焦距并采集眼底图像。此步骤在眼底相机镜头对准瞳孔并且达到工作距离时被执行，此时的镜头与眼球的位置如图13所示。需要说明的是，在上述实施例调整镜头位置和工作距离的过程中采集图像时，当然也需要设置固定的焦距，比如在调整工作距离时，焦距可以是固定调到0屈光位置。如果被拍摄者屈光正常，当工作距离调到位后，即可直接拍摄眼底图像。但实际应用时需要考虑被拍者的实际屈光度，从而设置适合的焦距。
在眼底相机进行曝光拍摄眼底图像之前，比如上述自动对准和自动确定工作距离的过程中都是用红外光来成像，此时采集图像所使用的光源仍然是红外光。虽然当前的焦 距不能使眼底清晰地成像，但在此时采集的图像已经能基本体现出眼底的特征，图像中至少可以显示出视盘，因此称采集的图像为眼底图像。
S2，在眼底图像中识别视盘区域。因为视盘区域是眼底中纹理最多、亮度最高的区域，因此最适合用于调焦。
在眼底图像中识别视盘的方式有多种，例如可以使用机器视觉算法，根据眼底图像中的图形特征检测视盘轮廓和位置。然而，由于使用红外光成像相对模糊，给识别视盘带来很大的挑战，计算机视觉算法遇到这种情况很容易误判，因此在一个优选的实施例中使用深度学习算法来解决这个问题。
首先采集大量眼底图像，这些图像对不同的人、使用不同焦距所采集的眼底图像。然后对每一张图像中的视盘进行标注，由此得到用于训练神经网络的训练数据。使用这些标注好的数据来训练一个神经网络模型(比如YOLO网络)，经过训练后，神经网络模型的识别结果包括一个检测框，用于表征眼底图像中视盘的位置。
如图14所示，在一个具体的实施例中，训练数据中采用方形框141来标注视盘，训练后的神经网络模型的识别结果也将是方形检测框。在其它实施例中，也可以使用圆形框进行标注，或者其它类似的标注方式都是可行的。
S3，根据视盘区域的清晰度确定拍摄焦距。具体地，可以从初始的焦距开始，采用梯度上升的方式不断变换焦距并采集相应的眼底图像，判断其中的视盘的清晰度是否达到预设标准，一旦达到预设标准，则判定当前的焦距为最佳焦距，不需要再继续搜索；也可以在焦距可调范围内使用所有可用的焦距，并采集相应的眼底图像，从所有眼底图像中确定一个视盘清晰度最高的眼底图像，判定采集该图像时的焦距为最佳焦距。
在一个具体的实施例中，采用遍历的方式，先在设定焦距范围800-1300内以第一设定步长40调整焦距并采集第一组眼底图像，由此到焦距800时的眼底图像、焦距840时的眼底图像、焦距880时的眼底图像……焦距1300时的眼底图像。分别在这些眼底图像中识别视盘区域，并分别确定每个眼底图像的清晰度，在本实施例中是计算视盘区域内的像素值的均值作为清晰度。然后可以从第一组眼底图像中确定一个具有最高清晰度的眼底图像，此时可以将采集该眼底图像时所使用的焦距X(第一焦距)作为拍摄焦距。
为了取得更好的拍摄效果，还可进一步搜索焦距，比如在上述焦距X附近再进行一次遍历，在此次遍历过程中所使用的第二设定步长小于上述第一设定步长，比如第二设定步长为10，由此可以进一步得到第二组眼底图像，即焦距X+10时的眼底图像、焦距X+20时的眼底图像、X-10时的眼底图像、X-20时的眼底图像等等。然后再分别在这些 眼底图像中识别视盘区域，并分别确定每个眼底图像的清晰度，比如确定焦距X-20时的眼底图像为清晰度最高的眼底图像，则将焦距X-20(第二焦距)作为拍摄焦距。
关于进一步搜索焦距的范围，作为优选的实施例，可将第一焦距X为中点，以增大第一设定步长为最大值、以减小第一设定步长为最小值的焦距范围,范围是X±40。
根据本发明实施例提供的焦距调整方法，在不同焦距下采集眼底图像，通过眼底图像中的视盘清晰度判断当前的焦距是否适用于拍摄眼底图像，不需要在眼底相机上设置任何额外的光学或者硬件，只需要设置图像识别算法即可找到最佳的对焦位置，由此可以降低眼底相机的成本，并提高焦距调整效率。
考虑到调整焦距的过程中用户可能会轻微转动头部等等，这将导致镜头不再是对准瞳孔的状态，因此在调整焦距的过程中，还将在XY平面上调整镜头的位置以保持对准瞳孔。并且，进行到此阶段时已经即将要拍摄眼底图像，如果被拍摄者在此时眨眼或者闭眼将无法成功拍摄，所以在此过程中还需要进行眨眼和/或闭眼的检测。本实施例提供一种优选的焦距调整方法，该方法包括如下步骤：
S1A，利用当前的焦距采集眼底图像。
S2A，通过眼底图像判断被拍摄者是否眨眼和/或闭眼。当被拍摄者眨眼和/或闭眼时进行提示，比如通过语音提示用户不要眨眼或闭眼等等,然后返回步骤S1A；否则执行步骤S3A。眨眼和闭眼检测也可以通过机器视觉算法或者神经网络算法实现，当被拍摄者眨眼或者闭眼时，采集的图像将是全黑或者非常模糊的，特征相对明显，可以采用多种方法进行检测，此处不再赘述。
S3A，识别眼底图像中是否存在由角膜所反射的照明光束形成的光斑。与上述实施例调整工作距离时保持镜头对准瞳孔的方式不同，在达到工作距离后，如果是对准的状态，角膜反射的照明光束应当不在成像范围内，眼底图像中不应该再出现上述光斑，尤其是不可能出现光斑的完整成像。即使出现光斑也将是整个光斑的一部分，在一个具体实施例中使用由多个照明灯呈环状排列形成的光源，完整的光斑如图12所示。如果在调整焦距时眼底图像中出现光斑，将是如图15所示的情况，其中只有部分光斑151。如果光源本身是一个完整的环形灯，则在图像中此出现带状物。
当眼底图像中存在光斑时，执行步骤S4A，否则执行步骤S5A。
S4A，至少根据光斑的位置微调镜头位置，以移除光斑，从而保持镜头对准瞳孔。当光斑出现在不同位置时，其尺寸和亮度会有所不同。作为优选的实施例，结合光斑在图像中的位置、尺寸和亮度可计算出矢量偏移。以图15为例，将图像中心作为原点(0,0)建立坐标系，图像半径为R。计算各个光斑151的近似圆形区域，在本实施例中近似圆 区域是包含光斑151的最小圆形区域。比如第i个光斑的近似圆形区域的中心坐标为(x i,y i),半径是r i。那么可以得出第i个光斑需要移动的方向是v i＝(x i,y i),需要移动的距离是
Figure PCTCN2021073875-appb-000001
其中k＝x i 2+y i 2，进而得出当前光斑需要移动v im i，将所有光斑需要移动的量进行求和，得到镜头需要移动的矢量152为∑vm。
再次使镜头对准瞳孔后返回步骤S1A。
S5A，在眼底图像中识别视盘区域，判断视盘区域的清晰度是否达到设定标准。在本实施例中使用mobilenet-yolov3神经网络模型识别视盘，神经网络输出的视盘区域是包含视盘和背景的区域。然后通过边缘检测算法(如sobel、Laplace等算法)在此视盘区域内检测视盘的边缘，得到准确的视盘图像，并计算视盘图像的均值作为清晰度值。
比如可以通过将得到的清晰度值与阈值进行比较来判断是否达到设定标准，如果视盘区域的清晰度未达到设定标准，执行步骤S6A。如果视盘区域的清晰度已达到设定标准，则判定当前的焦距适合拍摄眼底图像，之后即可关闭红外光，使用白光进行曝光，拍摄眼底图像。
S6A，调整焦距，之后返回步骤S1A。根据步骤S1A中所使用的初始焦距，比如初始焦距为可调焦距中的最小值，此时则按照固定步长或可变步长增大焦距，反之则减小焦距。
在利用上述各个实施例提供的方案将镜头对准瞳孔、调整至最佳工作距离并确定焦距后，开始拍摄眼底图像。在拍摄眼底图像时，需要使用照明组件进行曝光(本实施例的相机所使用的光源为白光)。然而在曝光拍摄过程中，被拍者仍有可能影响眼底图像的拍摄质量，比如瞳孔变小、眼皮遮挡、眨眼、面贴组件漏光等，当出现这些情况时所拍摄的眼底图像将出现不可用的区域。为了提高拍摄成功率，针对上述步骤S700，本实施例提供一种眼底图像拍摄方法，该方法可以由眼底相机本身来执行，也可以由计算机或者服务器等电子设备执行(作为一种控制方法)，该方法包括如下步骤：
S1，保持镜头状态不变并拍摄多个眼底图像。具体是指，根据上述各个实施例的方法将镜头固定在XY平面内的位置上对准瞳孔，并且定位在Z轴的距离上，使用固定的焦距，在镜头位置、工作距离和焦距保持不变的情况下，使照明组件曝光并拍摄多个眼底图像。
S2，分别确定多个眼底图像的质量。分析眼底图像质量的手段有多种，比如可以参考中国专利文件CN108346149A中提供的对于眼底图像的检测方法。在本实施例中，使 用神经网络模型来分析图像质量，神经网络模型可以执行分类任务，对图像质量进行分类，比如输出质量高或质量差的分类结果；也可以执行回归预测任务，对图像质量进行量化，比如输出1-10分来表达对图像质量的评价。
关于模型的训练，预先采集大量白光曝光的视网膜图片，人工标注图像质量为好或者不好(适用于分类模型)，或者给图像质量打分(例如1到10分，适用于回归预测模型)。将这些眼底图像和标注或者评分作为训练数据训练神经网络模型，模型收敛后即可用于识别眼底图像的质量。
S3，判断各个眼底图像的质量是否达到设定标准，如果有任意一个眼底图像达到设定标准，则将该眼底图像作为拍摄结果即可(输出拍摄结果)。如果多个眼底图像的质量全部未达到设定标准，则执行步骤S4。
S4，利用多个眼底图像合成一个眼底图像作为拍摄结果。连续拍照的多张眼底图像可能每一张整体质量都不好，但每一张都有可能有一部分质量较好的区域，利用这些可用的区域进行拼接和融合，即可得到一张高质量且完整的眼底图像。
根据本发明实施例提供的眼底图像拍摄方法，在保持镜头状态不变并拍摄多个眼底图像，并分别确定多个眼底图像的质量，当判定所有眼底图像都不可用时，利用多个眼底图像合成一个完整的眼底图像，即使被拍者干扰了拍摄过程，也将可能地利用现有的眼底图像得到质量较高的眼底图像，减少重新拍摄的次数，降低了用户的使用难度，提高了拍摄眼底图像的成功率。
进一步地，本发明实施例提供一种眼底图像合成方法,该方法包括如下步骤：
S41，获取镜头状态不变的情况下拍摄的多个眼底图像。这些眼底图像分别存在质量较差的区域和质量较好的区域。当然，如果某些眼底图像质量极差，比如评分为0的图像可能是全黑或者全白的，可以直接将这些完全不可用的图像去除。
S42，分别在多个眼底图像中提取高质量区域。在此步骤中，可以根据眼底图像的像素值计算亮度，通过与亮度阈值进行比较，去除亮度较高的区域以及亮度较低的区域，由此来去除曝光过度和曝光不足的区域，从而提取到亮度适中的区域，即高质量区域；也可以根据眼底图像的像素值计算锐度，通过与锐度阈值进行比较，去除锐度较低的区域，由此来去除曝模糊区域，从而得到高质量区域；或者根据亮度和锐度综合提取高质量区域。
根据眼底图像的实际亮度和/或锐度提取到的区域通常是边界不规则的区域，比如图16所示的两个高质量区域，左侧所示区域来自一个眼底图像的上部、右侧所示区域来自一个眼底图像的下部。
在其它可选实施例中，也可以按照固定的划分方式将各个眼底图像划分为网格，然后分别分析各个网格区域的质量，提取高质量的网格，这样可以得到边界规则的高质量区域。
S43，利用多个高质量区域合成眼底图像。由于各个眼底图像可能会存在一些偏移，为了更准确地合成眼底图像，在此可以先根据偏移量将各个眼底图像映射到同一坐标系下再进行拼接与融合处理。
作为优选的实施例，如图17所示，首先对多个眼底图像进行异常区域检测，以提取高质量区域。在步骤S43中，首先分别对多个眼底图像进行特征点提取(或称为关键点)，可以是视盘的中心点、血管的交叉点到等显著的位置。之后进行特征点匹配，去匹配不同眼底图像之间的特征点，这些特征点匹配之后，匹配信息用来计算各个眼底图像之间的偏移量(投影矩阵计算)。再根据偏移量将多个高质量区域映射为一张眼底图像。对于多个高质量区域之间存在的重叠部分，比如图16所示的两个区域，它们的中部是重复的，可以利用多个高质量区域的像素值和相应的权重确定重叠部分的像素值。这是一种基于加权平均的融合处理，作为举例，该融合处理可以表示为q1/(q1+q2)*image1+q2/(q1+q2)*image2，其中q1表示对应于第一个高质量区域的权重、q2表示对应于第二个高质量区域的权重，image1表示第一个高质量区域、image2表示第二个高质量区域。
上述权重的取值根据眼底图像的整体质量进行设置，比如第一个高质量区域取自第一眼底图像、第二个高质量区域取自第二眼底图像，而根据上述质量分析方法所得到的第一眼底图像的质量(比如神经网络输出的分值)比第二眼底图像的质量更高，那么相应的权重q1大于q2。
如图16、17所展示的情况只是为了说明本方案的原理所做的举例，实际使用时将会拍摄更多的眼底图像，尽可能确保提取到更多的高质量区域，以保证生成的眼底图像完整。
根据本发明实施例提供的眼底图像合成方法，当对被拍者的所拍摄的多个眼底图像都存在瑕疵时，利用本方案分别在多个眼底图像中提取高质量区域，进行拼接和融合得可以得到质量较高的完整眼底图像，由此降低用户自拍眼底图像的难度，提高了拍摄成功率。
本领域内的技术人员应明白，本发明的实施例可提供为方法、系统、或计算机程序产品。因此，本发明可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的 计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本发明是参照根据本发明实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
显然，上述实施例仅仅是为清楚地说明所作的举例，而并非对实施方式的限定。对于所属领域的普通技术人员来说，在上述说明的基础上还可以做出其它不同形式的变化或变动。这里无需也无法对所有的实施方式予以穷举。而由此所引伸出的显而易见的变化或变动仍处于本发明创造的保护范围之中。

Claims (11)

  1. 一种眼底图像全自动拍摄方法，其特征在于，包括：
    移动眼底相机镜头对准瞳孔；
    控制所述镜头接近眼球并采集图像，所述图像是对角膜所反射的照明光束的成像；
    利用所述图像确定工作距离；
    调整焦距并采集眼底图像，利用所述眼底图像确定拍摄焦距；
    在所述工作距离上使用所述拍摄焦距拍摄眼底图像。
  2. 根据权利要求1所述的方法，其特征在于，在移动眼底相机镜头对准瞳孔之前，还包括：检测眼底相机的运动组件、照明组件和调焦组件是否正常。
  3. 根据权利要求2所述的方法，其特征在于，检测眼底相机的运动组件、照明组件和调焦组件是否正常具体包括：
    控制运动组件调节镜头的位置，检测所述镜头是否能够移动到各个定位组件所在的位置；
    在所述镜头能够移动到各个定位组件所在的位置后，控制所述运动组件将所述镜头移动到设定位置，开启照明组件并控制调焦组件调整为第一焦距，拍摄得到第一图像；
    根据所述第一图像中的照明组件的图像特征判断所述调焦组件和所述照明组件是否正常；
    当所述调焦组件和所述照明组件正常时，控制运动组件调节镜头至设定深度位置上，控制调焦组件调整为第二焦距，拍摄得到第二图像；
    根据所述第二图像中的被拍物的图像特征判断成像功能是否正常。
  4. 根据权利要求1或2所述的方法，其特征在于，在移动眼底相机镜头对准瞳孔之前，还包括：检测人体头部是否贴合眼底相机的面贴组件。
  5. 根据权利要求4所述的方法，其特征在于，检测人体头部是否贴合眼底相机的面贴组件具体包括：
    关闭照明组件，获取镜头透过面贴组件的视窗采集的第一图像；
    判断所述第一图像的亮度是否达到设定标准；
    当所述第一图像的亮度达到设定标准时，开启所述照明组件，获取镜头透过面贴组件的视窗采集的第二图像；
    根据所述第二图像确定人体头部是否贴合所述面贴组件。
  6. 根据权利要求1-5中任一项所述的方法，其特征在于，利用所述图像确定工作距离具体包括：
    检测所述图像中的光斑的特征是否符合设定特征；
    当所述光斑的特征符合设定特征时，确定达到工作距离。
  7. 根据权利要求1-5中任一项所述的方法，其特征在于，利用所述眼底图像确定拍摄焦距具体包括：
    在所述眼底图像中识别视盘区域；
    根据所述视盘区域的清晰度确定拍摄焦距。
  8. 根据权利要求1-5中任一项所述的方法，其特征在于，在所述工作距离上使用所述拍摄焦距拍摄眼底图像具体包括：
    判断瞳孔尺寸是否小于眼底相机照明组件的环状照明光束尺寸；
    当所述瞳孔尺寸小于所述环状照明光束尺寸时，分别向多个方向移动所述镜头与瞳孔产生偏移，使得所述环状照明光束部分照射在瞳孔中，并拍摄多个眼底图像；
    将所述多个眼底图像融合为一个眼底图像。
  9. 根据权利要求1-5中任一项所述的方法，其特征在于，在所述工作距离上使用所述拍摄焦距拍摄眼底图像具体包括：
    获取镜头状态不变的情况下拍摄的多个眼底图像；
    分别在所述多个眼底图像中提取高质量区域；
    利用多个所述高质量区域合成眼底图像。
  10. 一种电子设备，其特征在于，包括：至少一个处理器；以及与所述至少一个处理器通信连接的存储器；其中，所述存储器存储有可被所述一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器执行如权利要求1-9中任一项所述的眼底图像全自动拍摄方法。
  11. 一种眼底相机，其特征在于，包括：面贴组件、运动组件、调焦组件、照明组件、镜头和至少一个处理器；以及与所述至少一个处理器通信连接的存储器；其中，所述存储器存储有可被所述一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器执行如权利要求1-9中任一项所述的眼底图像全自动拍摄方法。
PCT/CN2021/073875 2020-10-14 2021-01-27 眼底相机及眼底图像全自动拍摄方法 WO2022077800A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP21878859.4A EP4230112A1 (en) 2020-10-14 2021-01-27 Fundus camera and fully-automatic photography method for fundus image
US18/031,513 US20230404401A1 (en) 2020-10-14 2021-01-27 Fundus camera and fully-automatic photography method for fundus image
JP2023519780A JP2023547595A (ja) 2020-10-14 2021-01-27 眼底カメラ及び眼底画像全自動撮影方法

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011095133.3A CN112043236B (zh) 2020-10-14 2020-10-14 眼底相机及眼底图像全自动拍摄方法
CN202011095133.3 2020-10-14

Publications (1)

Publication Number Publication Date
WO2022077800A1 true WO2022077800A1 (zh) 2022-04-21

Family

ID=73605181

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/073875 WO2022077800A1 (zh) 2020-10-14 2021-01-27 眼底相机及眼底图像全自动拍摄方法

Country Status (5)

Country Link
US (1) US20230404401A1 (zh)
EP (1) EP4230112A1 (zh)
JP (1) JP2023547595A (zh)
CN (1) CN112043236B (zh)
WO (1) WO2022077800A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116369840A (zh) * 2023-06-05 2023-07-04 广东麦特维逊医学研究发展有限公司 一种无亮斑投影照明系统及其工作方法
CN116725479A (zh) * 2023-08-14 2023-09-12 杭州目乐医疗科技股份有限公司 一种自助式验光仪以及自助验光方法
CN117893529A (zh) * 2024-03-14 2024-04-16 江苏富翰医疗产业发展有限公司 一种眼底智能拍摄方法

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112043236B (zh) * 2020-10-14 2021-06-15 上海鹰瞳医疗科技有限公司 眼底相机及眼底图像全自动拍摄方法
CN113476014B (zh) * 2021-06-02 2023-11-14 北京鹰瞳科技发展股份有限公司 双坐标系对应关系建立系统及方法
CN113349734B (zh) * 2021-06-29 2023-11-14 北京鹰瞳科技发展股份有限公司 眼底相机及其工作距离校准方法
CN113729617A (zh) * 2021-08-20 2021-12-03 北京鹰瞳科技发展股份有限公司 眼底相机的镜头的控制方法和控制装置
CN114098632B (zh) * 2022-01-27 2022-11-29 北京鹰瞳科技发展股份有限公司 用于对眼底相机中的马达进行控制的方法及其相关产品
CN115471552B (zh) * 2022-09-15 2023-07-04 江苏至真健康科技有限公司 一种用于便携式免散瞳眼底照相机的拍摄定位方法及系统
CN116687339B (zh) * 2023-08-01 2023-10-31 杭州目乐医疗科技股份有限公司 基于眼底相机的图像拍摄方法、眼底相机、装置及介质

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005278842A (ja) * 2004-03-29 2005-10-13 Nidek Co Ltd 眼底カメラ
CN1714740A (zh) * 2004-06-14 2006-01-04 佳能株式会社 眼科装置
CN1810202A (zh) * 2004-12-21 2006-08-02 佳能株式会社 眼科装置
CN103908221A (zh) * 2013-01-08 2014-07-09 华晶科技股份有限公司 摄像装置及摄像方法
CN108346149A (zh) 2018-03-02 2018-07-31 北京郁金香伙伴科技有限公司 图像检测、处理方法、装置及终端
CN109547677A (zh) * 2018-12-06 2019-03-29 代黎明 眼底图像拍摄方法和系统及设备
CN111134616A (zh) 2020-02-25 2020-05-12 上海鹰瞳医疗科技有限公司 一种眼底相机照明系统及眼底相机
CN111449620A (zh) * 2020-04-30 2020-07-28 上海美沃精密仪器股份有限公司 一种全自动眼底相机及其自动照相方法
CN112043236A (zh) * 2020-10-14 2020-12-08 上海鹰瞳医疗科技有限公司 眼底相机及眼底图像全自动拍摄方法

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005278842A (ja) * 2004-03-29 2005-10-13 Nidek Co Ltd 眼底カメラ
CN1714740A (zh) * 2004-06-14 2006-01-04 佳能株式会社 眼科装置
CN1810202A (zh) * 2004-12-21 2006-08-02 佳能株式会社 眼科装置
CN103908221A (zh) * 2013-01-08 2014-07-09 华晶科技股份有限公司 摄像装置及摄像方法
CN108346149A (zh) 2018-03-02 2018-07-31 北京郁金香伙伴科技有限公司 图像检测、处理方法、装置及终端
CN109547677A (zh) * 2018-12-06 2019-03-29 代黎明 眼底图像拍摄方法和系统及设备
CN111134616A (zh) 2020-02-25 2020-05-12 上海鹰瞳医疗科技有限公司 一种眼底相机照明系统及眼底相机
CN111449620A (zh) * 2020-04-30 2020-07-28 上海美沃精密仪器股份有限公司 一种全自动眼底相机及其自动照相方法
CN112043236A (zh) * 2020-10-14 2020-12-08 上海鹰瞳医疗科技有限公司 眼底相机及眼底图像全自动拍摄方法

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116369840A (zh) * 2023-06-05 2023-07-04 广东麦特维逊医学研究发展有限公司 一种无亮斑投影照明系统及其工作方法
CN116369840B (zh) * 2023-06-05 2023-08-01 广东麦特维逊医学研究发展有限公司 一种无亮斑投影照明系统及其工作方法
CN116725479A (zh) * 2023-08-14 2023-09-12 杭州目乐医疗科技股份有限公司 一种自助式验光仪以及自助验光方法
CN116725479B (zh) * 2023-08-14 2023-11-10 杭州目乐医疗科技股份有限公司 一种自助式验光仪以及自助验光方法
CN117893529A (zh) * 2024-03-14 2024-04-16 江苏富翰医疗产业发展有限公司 一种眼底智能拍摄方法

Also Published As

Publication number Publication date
JP2023547595A (ja) 2023-11-13
CN112043236A (zh) 2020-12-08
EP4230112A1 (en) 2023-08-23
CN112043236B (zh) 2021-06-15
US20230404401A1 (en) 2023-12-21

Similar Documents

Publication Publication Date Title
WO2022077800A1 (zh) 眼底相机及眼底图像全自动拍摄方法
CN112220447B (zh) 眼底相机及眼底图像拍摄方法
CN112075921B (zh) 眼底相机及其焦距调整方法
US7845797B2 (en) Custom eyeglass manufacturing method
US7434931B2 (en) Custom eyeglass manufacturing method
US8644562B2 (en) Multimodal ocular biometric system and methods
JP2000113199A (ja) 眼の位置を判断する方法
CN111449620A (zh) 一种全自动眼底相机及其自动照相方法
CN106821303B (zh) 一种自主式眼底摄影成像系统及方法
CN112075920B (zh) 眼底相机及其工作距离调整方法
CN111202495A (zh) 皮肤分析装置
WO2021204211A1 (zh) 人脸和虹膜图像采集方法、装置、可读存储介质及设备
JP5092120B2 (ja) 眼球運動計測装置
CN112190227B (zh) 眼底相机及其使用状态检测方法
JP2017034569A (ja) 撮像装置及びその制御方法
CN112220448B (zh) 眼底相机及眼底图像合成方法
CN112043237A (zh) 全自动便携自拍眼底相机
JP3711053B2 (ja) 視線測定装置及びその方法と、視線測定プログラム及びそのプログラムを記録した記録媒体
CN112190228B (zh) 眼底相机及其检测方法
Borsato et al. Episcleral surface tracking: challenges and possibilities for using mice sensors for wearable eye tracking
CN212281326U (zh) 全自动便携自拍眼底相机
CN113658243B (zh) 眼底三维模型建立方法、眼底照相机、装置和存储介质
KR102263830B1 (ko) 자동 초점 기능을 이용한 안저 영상 촬영 장치
KR102085285B1 (ko) 딥러닝 영상분석 기반의 얼굴 인식 및 홍채 위치 인식 시스템
EP3695775B1 (en) Smartphone-based handheld optical device and method for capturing non-mydriatic retinal images

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21878859

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023519780

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 18031513

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2021878859

Country of ref document: EP

Effective date: 20230515