WO2020215361A1

WO2020215361A1 - Method and system for image stabilization based on improved line scan imaging system

Info

Publication number: WO2020215361A1
Application number: PCT/CN2019/085503
Authority: WO
Inventors: 张�杰; 张金莲
Original assignee: 南京博视医疗科技有限公司
Priority date: 2019-04-25
Filing date: 2019-05-05
Publication date: 2020-10-29
Also published as: CN109924943A

Abstract

Disclosed are a method and a system for image stabilization based on an improved line scan imaging system, The system for image stabilization comprises a line scan ophthalmoscope (LSO) imaging system, the LSO imaging system comprising an orthogonal galvanometric scanner, wherein the orthogonal galvanometric scanner generates orthogonal scan surfaces at a fundus and is capable of adjusting the scanning surfaces to any position in a 360-degree space; and is configured for achieving fundus optical tracking inside the LSO optical system by means of an intelligent control algorithm while the LSO optical system performs two-dimensional fundus scanning. A closed-loop fundus optical tracking system can be established, and the line scan imaging system can be controlled using the closed-loop tracking system, thus achieving high-speed, stable and precise control.

Description

Image stabilization method and system based on improved line scan imaging system

Technical field

The invention relates to laser fundus target tracking and imaging technology, in particular to an image stabilization method and system based on an improved line scan imaging system.

Background technique

The existing target tracking technology based on the Line Scan Ophthalmoscope (LSO) imaging system, such as the Carl Zeiss imaging system, uses one frame of image as a unit to calculate the amount of movement of the fundus target. However, there is a defect that the control system has a time delay of at least one frame, which results in a decrease in tracking accuracy. Moreover, the target tracking signal method of the existing LSO is completely digital. When extracting the signal from the image, the lack of optical closed-loop control measures inside the LSO will also cause the calculation of the fundus movement signal to be unreliable.

Summary of the invention

In view of this, the main purpose of the present invention is to provide an image stabilization method and system based on an improved line scan imaging system, which aims to overcome the inherent optical and control defects of the existing LSO imaging system and greatly improve its clinical application Stability, accuracy, and imaging efficiency in the medium.

In order to achieve the above objective, the technical solution of the present invention is achieved as follows:

An image stabilization system based on an improved line scan imaging system, including a line scan fundus camera LSO imaging system. The LSO imaging system also includes an orthogonal galvanometer scanning device for generating an orthogonal scan surface on the fundus. The scanning surface is adjusted to any position in the 360-degree space; and used to combine with the intelligent control algorithm to realize the LSO optical system to perform two-dimensional fundus scanning while simultaneously performing the fundus optical tracking inside the LSO.

Wherein, the orthogonal galvanometer scanning device is a dual-mirror structure composed of a first mirror SM11 and a second mirror SM12, or a MEMS scanning mirror of a micro-electromechanical system that vibrates orthogonally and bidirectionally, or other orthogonal bidirectional vibrating mirrors composition.

The orthogonal galvanometer scanning device combined with an intelligent control algorithm realizes that the LSO optical system performs fundus optical tracking inside the LSO while scanning the two-dimensional fundus. The image stabilization system has the following relationship:

(x _t+1 ,y _t+1 )=(x _t ,y _t )+g(Δx _t ,Δy _t ) (1)

Among them, (x _t , y _t ) represents the control commands on the first mirror SM11 and the second mirror SM12 at the current sampling moment, which are equivalent to their respective motion offsets; (Δx _t , Δy _t ) represents the slave The relative motion of the image target frame and the reference frame recorded by the line scan camera; g represents the gain of the closed-loop control system; (x _t+1 ,y _t+1 ) represents the existing signal applied to the first mirror SM11 and the second reflection The next set of new commands of the mirror SM12 is equivalent to the movement offset.

An image stabilization method based on an improved line scan imaging system includes the following steps:

A. An orthogonal galvanometer scanning device is added to the online scanning fundus camera LSO imaging system. The orthogonal galvanometer scanning device is used to generate an orthogonal scanning surface on the fundus, so that the scanning surface can be adjusted to any position in a 360-degree space ；

B. Combining the orthogonal galvanometer scanning device with an intelligent control algorithm to realize the LSO optical system to perform the fundus optical tracking inside the LSO while scanning the two-dimensional fundus.

An image stabilization system based on an improved line scan imaging system, further comprising a rotating device for setting the cylindrical mirror L13 that generates the line light source and the line scan camera coupled with it on a 360-degree space controllable rotating bracket, Make the line controllable light source rotate at any position in 360-degree space.

As an embodiment, the sawtooth wave is used as the driving signal base of the first mirror SM11 and the second mirror SM12, and the rotation angle is set to θ, and the amplitude due to each mirror is multiplied by the respective base signal, then Equation (1) can be updated as:

(x _t+1 ,y _t+1 ,θ _t+1 )=(x _t ,y _t ,θ _t )+g(Δx _t ,Δy _t ,Δθ _t ) (2)

Among them, θ _t is the angle imposed by the closed-loop control system on the rotating support; (x _t ,y _t ) is the translational amount imposed on the first mirror SM11 and the second mirror SM12, (x _t ,y _t ) It is still superimposed on the translation amount of each mirror used to generate the scanning signal; (x _t , y _t , θ _t ) is the current sampling time on the first mirror SM11 and the second mirror SM12 and the rotating bracket Control instructions, which are equivalent to their respective movement offsets and rotation angles; (Δx _t , Δy _t , Δθ _t ) is the relative movement of the image target frame and the reference frame recorded from the line scan camera; g is the closed-loop control system (X _t+1 ,y _t+1 ,θ _t+1 ) is the current signal applied to the first mirror SM11 and the second mirror SM12 as well as the cylindrical mirror L13 and the line scan camera coupled with it The next set of new instructions for rotating the bracket is equivalent to the movement offset and rotation angle.

An image stabilization method of an image stabilization system based on an improved line scan imaging system includes the following steps:

B. Mount the cylindrical mirror L13 that generates the line light source and the line scan camera coupled with it on a 360-degree controllable rotating bracket, so that the line-expanded light source can be rotated at any position in the 360-degree space;

C. Combining the orthogonal galvanometer scanning device with an intelligent control algorithm to realize the LSO optical system to perform the fundus optical tracking inside the LSO while scanning the two-dimensional fundus.

The present invention is based on an improved image stabilization method and system of a line scan imaging system, and has the following beneficial effects:

1) By establishing a closed-loop fundus optical tracking system inside the LSO imaging system, the closed-loop tracking control system is used to control the line scan imaging system, so as to achieve the purpose of high-speed, stable and precise control.

2) The LSO closed-loop control system can be used to obtain the fundus movement signal, and through the pre-calibrated spatial transformation relationship, to control another one or more optical systems to achieve the corresponding fundus target tracking purpose.

3) According to the order in which each scan line arrives at the host system in each frame of the image, a frame of image is divided into multiple sub-frame elements in chronological order, and each sub-frame element contains one or more scan lines . According to the order in which each sub-frame element arrives at the host system, the fundus movement information contained in each sub-frame element is calculated in real time, and then immediately fed back to the tracking device, such as a high-speed tilt mirror and a rotating table. Using this frequency multiplication method can greatly improve the spatial accuracy and time bandwidth of target tracking.

Description of the drawings

Figure 1 is a schematic diagram of the optical structure of a conventional line scan fundus camera;

Figure 2 is a schematic diagram of a sawtooth wave used to control the scanning mirror SM;

3 is a schematic diagram of a fundus image obtained according to the optical system of the line scan fundus camera shown in FIG. 1;

Figure 4 is a schematic diagram of an existing line scan imaging system, including a main LSO imaging system without optical tracking integrated with a secondary OCT imaging system;

Fig. 5 is a schematic diagram of calculating the amount of fundus movement from an image in a frame unit obtained by the image stabilization system based on the line scan imaging system of the present invention;

6 is a schematic diagram of an improved LSO optical system with internal optical tracking according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of the working state of two inclined mirrors SM11 and SM12 in the improved LSO optical system shown in FIG. 6;

FIG. 8 is a schematic diagram of adjusting the position of the imaging surface in a 360-degree space by changing the offset of the mirrors SM11 and SM12;

9 is a schematic diagram of the position of the line light source generated by the rotating cylindrical lens L13 and the line scan camera coupled with the rotating device in a 360-degree space;

FIG. 10 is a schematic diagram of a state where a rotating cylindrical mirror generates a line light source with an arbitrary rotation angle and a scanning surface related to it;

11 is a schematic diagram of the principle of integrating a main LSO imaging system with closed-loop optical tracking into another auxiliary imaging system according to an embodiment of the present invention;

12 is a schematic diagram of using frequency doubling technology to reduce the time delay of fundus calculation in an embodiment of the present invention;

FIG. 13 is a schematic diagram of the scanning (reflecting) mirror SM11 scanning signal and the division mode of sub-frame elements;

Figure 14 is a schematic diagram of scanning signals and synchronization signals of a line scanning system;

Figure 15 is a signal that combines a line reference clock and a frame synchronization signal to trigger a line scan camera.

Detailed ways

The present invention will be further described in detail below in conjunction with the drawings and embodiments of the present invention.

FIG. 1 is a schematic diagram of the optical structure of a conventional line scan fundus camera.

As shown in FIG. 1, the light emitted from the point light source L11 is collimated by the lens L12, the surface light source is converted into a line light source through a cylinder lens (Cylinder Lens) L13, and then it is relayed to the collimating lens L14. Here, the installation direction of the cylindrical lens L13 determines the expansion direction of the line light source in space (see Figures 9 and 10 for details), and the lens L12, lens L13, and lens L14 determine to a certain extent the line light source's illumination (expansion) in the fundus size. Part of the light emitted by the lens L14 passes through the beam splitter (BS) and reaches the scanning mirror (Steering Mirror or Scanning Mirror, SM); the other part passes through the beam splitter (BS) to the collimating lens L17, and then passes through a Group filter (FILTER), reach line scan camera (Line Scan Camera).

Among them, the function of the scanning mirror SM is to generate periodic scanning in the orthogonal direction of the line light source, and the light passes through two collimating zoom lenses L15 and L16 to generate a two-dimensional scanning space at the bottom of the eye (Eye). The motion track of the scanning mirror (SM) generally presents a sawtooth wave as shown in FIG. 2.

Figure 2 is a schematic diagram of a sawtooth wave used to control a scanning mirror (SM). The frequency of the sawtooth wave determines the image frame rate of the imaging system, and the amplitude of the sawtooth wave determines the size of the optical field of view in the scanning direction.

As shown in Figure 2, the center of the sawtooth wave is not always at the zero position of the sawtooth wave. The center offset of the sawtooth wave actually determines the center position of the scanning field of view. Within the range allowed by the optical design, users are supported to control the center position of the scanning field of view by adjusting the center offset of the sawtooth wave.

Referring to Figure 1, when the fundus is excited by the light emitted by the point light source L11, the returned signal is reflected from the beam splitter BS to the collimating lens L17 through the same optical path, and then passes through a set of filters (FILTER) to reach the line scan camera ( Line Scan Camera). Among them, the signal returned from the fundus may be a reflected signal, a fluorescent signal, or other signals; it may also be a variety of other signals that reach the line scan camera at the same time.

3 is a schematic diagram of a fundus image obtained according to the optical system of the line scan fundus camera shown in FIG. 1. That is, the schematic diagram of the fundus image obtained by the line scan fundus camera shown in FIG. 1.

Fig. 4 is a system schematic diagram of an existing line scan imaging system, including a main LSO imaging system without optical image stabilization (or tracking) and an integrated auxiliary optical coherence tomography (OCT) imaging system.

As shown in Figure 4, the main LSO imaging system is the imaging system shown in Figure 1. Preferably, when the main LSO imaging system is used clinically, it will be equipped with an auxiliary imaging system customized according to the embodiment of the present invention, such as the OCT product Cirrus of Carl Zeiss. The auxiliary imaging system shown in FIG. 4 is an OCT Device.

In the auxiliary imaging system shown in Figure 4, the light emitted from the second point light source L21 reaches the orthogonal scanning mirrors SM21 and SM22 through the collimating system (including the collimating lenses L22 and L23), and then it is focused to the splitting mirror through the focusing lens L24. On the Dichotic Mirror (DM). The DM is also located on the focal plane of the main LSO imaging system.

In the embodiment of the present invention, the main and auxiliary (integrated optical) imaging systems shown in FIG. 4 support simultaneous implementation of two-dimensional fundus reflection (or fluorescence) imaging and three-dimensional OCT tomographic imaging.

One function of the main LSO imaging system is to provide fundus positioning and navigation for the auxiliary imaging system, and to display the current OCT tomogram in the corresponding position of the fundus two-dimensional space to the current user. Another function of the main LSO imaging system is to calculate fundus/eye movement information (x, y, θ) obtained from the LSO image by executing a preset algorithm (see below for details). Among them, (x, y) is the amount of translation of fundus movement, and θ is the amount of rotation. Then, (x, y, θ) is applied to the scanning mirrors SM21 and SM22 of the auxiliary imaging system, and the corresponding spatial positions of the scanning mirrors SM21 and SM22 are adjusted in real time to obtain the tomographic image of the required fundus position. However, the acquisition of (x, y, θ) here is completely digital, and is not achieved through the optical closed loop described in equation (1) or equation (2).

The above-mentioned fundus positioning and navigation process, as well as fundus tracking technology, through the main LSO image, apply cross-correlation algorithm (cross correlation) or other similar algorithms to calculate the fundus movement position (x, y, θ). Adjust the optical scanning position of the scanning mirrors SM21 and SM22 in real time to lock the fundus target. The acquisition of (x, y, θ) here is entirely digital, not through the optical closed loop described in equation (1) or equation (2).

The above-mentioned fundus tracking technology has the following characteristics:

First, the main LSO system simply obtains an image similar to Fig. 3 and uses cross-correlation or similar algorithms to calculate digital fundus motion information (x, y, θ). The acquisition of (x, y, θ) is entirely digital, not through the optical closed loop described in equation (1) or equation (2).

Second, the fundus tracking only occurs on the scanning mirrors SM21 and SM22 of the auxiliary imaging system. The main LSO system does not adjust its own optical parameters to lock the LSO scanning (imaging) position of the fundus accordingly.

Third, the accuracy and reliability of the numerical calculation results (x, y, θ) here are largely dependent on various parameters, including fundus image quality, normal fundus movements including blinking and saccade ) And micro saccade. For example, in the cross-correlation algorithm, when the target image (the amount of movement to be calculated) drifts out of the reference image, that is, when the amount of eye movement is too large, the cross-correlation algorithm cannot obtain accurate fundus movement information, which will lead to the tracking of the auxiliary imaging system failure.

Fourth, the prior art calculation (x, y, θ) is based on a frame, as shown in FIG. 5, which is a schematic diagram of an image obtained by calculating the amount of fundus movement from an image in a frame unit.

Referring to FIG. 5, it is assumed that f ₁ is the first frame image captured by the LSO, and f ₁ is used and defined as a "reference frame" image. In the time series, the images obtained after the system are f ₂ , f ₃ , ..., f _n , f _n+1 , which are defined as "target frame" images.

In the existing fundus tracking technology, the LSO software program usually starts the cross correlation algorithm (Cross Correlation) after receiving a complete image frame f _k (k=2,3,4,...,n+1) Calculate the spatial position of f _k relative to f ₁ (x _k , y _k , θ _k ). Once the algorithm program obtains (x _k , y _k , θ _k ), it is immediately converted to the scanning mirrors SM21 and SM22 of the auxiliary imaging system through the predetermined spatial mapping relationship, so that the scanning mirrors SM21 and SM22 are locked in the required The position of the fundus scan.

However, this frame-based calculation method, due to the large time delay, using (x _k , y _k , θ _k ) to control the positions of the scanning mirrors SM21 and SM22 will bring about a large spatial error, that is, the tracking space The accuracy is not high (tens to hundreds of microns) and the time response is slow. The reason is that a typical imaging system outputs 25 to 30 frames of images per second, so the time delay carried by each frame of image is already 33 to 40 milliseconds.

For example, the prerequisite for applying the cross correlation algorithm (Cross Correlation) to calculate the amount of eye movement from the image is that the image is needed. As mentioned above, it takes 33-40 milliseconds to acquire a frame of image, plus the time of the algorithm, the (x _k , y _k , θ _k ) obtained from the algorithm is converted into the electronic delay of the control signals of the scanning mirrors SM21 and SM22, and then The scanning mirrors SM21 and SM22 themselves respond to the mechanical delay of the control signal. A complete control cycle, from the start of the eye movement to the scanning mirror SM21 and SM22 tracking the movement, the delay time reaches 40-50 milliseconds is a very common phenomenon. From the above analysis process, it can be seen that among all the factors that can cause delay, the (image) sampling delay of 33-40 milliseconds is usually the dominant latency.

Correspondingly, one way to shorten the above-mentioned time delay is to greatly increase the frame rate of image output, for example, LSO outputs 200 frames per second, so that the delay of image sampling can be reduced to 5 milliseconds. However, to maintain the same image signal-to-noise ratio in the same imaging field of view, the side effect of the increase in the frame rate of the imaging system is the rapid increase in the nonlinearity of the imaging laser dose. This is not clinically feasible because the use of laser doses is restricted by safety standards.

In summary, the existing LSO imaging systems (products) have deficiencies in optics, electronics, and control, so that the present invention is capable of optics, electronics, software, and control based on the systems of Figures 1 and 4 Further improvements and enhancements.

Fig. 6 is a schematic diagram of an improved LSO optical system according to an embodiment of the present invention.

As shown in Figure 6, on the traditional LSO optical system shown in Figure 1, a second tilted mirror is added. As another embodiment, the two one-dimensional galvanometers (SM11 and SM12) in FIG. 6 can also use a Microelectro Mechanical Systems (MEMS) scanning mirror with orthogonal bidirectional vibration, or other orthogonal scanning reflectors. Mirror replacement.

In Fig. 6, the difference from Fig. 1 is that a tilting (reflecting) mirror is added. The mirror (SM) in Fig. 1 is named the first mirror SM11, and the newly added mirror is named the second mirror SM12 . The working process of the mirrors SM11 and SM12 is shown in FIG. 7.

Here, the dual-mirror structure composed of the first mirror SM11 and the second mirror SM12, or a bidirectionally vibrating micro-electromechanical system (MEMS) scanning mirror, or other forms of orthogonal scanning mirror structures can be collectively referred to as orthogonal Galvo scanning device.

FIG. 7 is a schematic diagram of the working state of two inclined mirrors SM11 and SM12 in the improved LSO optical system shown in FIG. 6.

For ease of description, first define a spatial reference coordinate (x, y, z), as shown in Figure 7A. When there is only the mirror SM11 (SM1), referring to FIG. 7B, a line light source A is incident on the mirror SM11. Here, the rotation axis of the mirror SM11 is on the x-axis of the spatial coordinates, so that the mirror SM11 swings on the y-z plane, and then a two-dimensional scanning surface is generated at the position B. Referring to the conventional LSO imaging system shown in Figure 1, the position of B is directly conjugated to the imaging surface of the fundus.

However, in the embodiment of the present invention, after the linear light source from position A passes through the mirror SM11, a second tilting (reflecting) mirror SM12 is inserted at the position B in FIG. 7C. Following the above definition, here the rotation axis of the mirror SM12 is on the z-axis and swings in the x-y plane.

It is understandable that the reference coordinates (x, y, z) in FIG. 7A can be defined arbitrarily, as long as the motion axis of the mirror SM11 and the motion axis of the mirror SM12 are orthogonal.

The working mode of this double mirror can be realized by the double mirror structure shown in Figure 6, for example, two Cambridge Technology one-dimensional 6210H galvanometers or 6220H galvanometers, or one with two independent orthogonal motions It can be realized by the tilt mirror of the axis, such as PI's S-335.2SH fast tilt mirror (Tip/Tilt Mirror).

The function and effect of the combined use of mirrors SM11 and SM12 as shown in Figure 6 and Figure 7 is that the scanning surface generated on the fundus of the LSO can be adjusted by changing the offset of the mirrors SM11 and SM12 within the allowable range of the optical system. Adjust the scanning surface to any position in the 360-degree space. This will be further described in Figure 8 below.

Fig. 8 is a schematic diagram of adjusting the position of the imaging surface in a 360-degree space by changing the offset of the mirrors SM11 and SM12.

As shown in Figure 8, the parameter of controlling the mirror SM12 in simple cases (refer to Figure 9 and the following for complex control situations) is a translation amount to adjust the horizontal position of the imaging surface, which can be used to track the target Movement in the horizontal direction. Here, there are generally multiple parameters for controlling the mirror SM11. On the one hand, the mirror SM11 is used for scanning, and on the other hand, for vertical translation of the imaging surface or target tracking (refer to FIG. 2).

Combining the functions of the mirrors SM11 and SM12, combined with the intelligent control algorithm, the LSO optical system can realize the fundus optical tracking inside the LSO while scanning the two-dimensional fundus. For related control and algorithm implementation, please refer to Figure 11 and the following content.

In summary, Figure 6 constitutes a complete closed-loop control system. The light from the point light source L11 reaches the fundus through the mirrors SM11 and SM12, which is a two-dimensional scanning space. The signal returned from the space where the fundus is scanned is scanned by the mirrors SM11 and SM12 again to reach the photodetector, here is a Line Scan Camera, which is used to record the image signal returned from the fundus.

In addition, the reason why Figure 6 of the present invention can constitute a complete closed-loop control system is because after the fundus tracking system is activated, the system has such a relationship:

(x _t+1 ,y _t+1 )=(x _t ,y _t )+g(Δx _t ,Δy _t ) (1)

In the above formula (1), (x _t , y _t ) represents the control commands on the mirrors SM11 and SM12 (equivalent to their respective movement offsets) at the current sampling time, and (Δx _t , Δy _t ) represents the slave The relative motion between the image (target frame) recorded by the line scan camera and the reference image (reference frame), g represents the gain of the closed-loop control system, (x _t+1 ,y _t+1 ) represents the existing signal applied to the mirror SM11 And the next set of new commands of SM12 (equivalent to motion offset).

Before entering the photodetector (the line scan camera here), the motion signal from the fundus has been optically compensated by mirrors SM11 and SM12, so the motion signal obtained from the photodetector is always a residual motion signal, which is equation (1)的(Δx _t ,Δy _t ).

The closed-loop control described above can also compensate for the rotation signal of the eyeball. One method is to install the cylindrical mirror L13 that produces the line light source in Figure 6 and the coupled line scan camera on a 360-degree controllable rotating bracket, so that the line-extended light source can be rotated at any position in the 360-degree space (reference Figure 9).

FIG. 9 is a schematic diagram of the position of the line light source generated by the rotating cylindrical mirror L13 in the 360-degree space by the rotating device/mechanism.

As shown in Figure 9, the axis of the cylindrical mirror and the coupled line scan camera (for simplicity, the line scan camera is not shown in the figure) are installed in a controllable rotating mechanism at the coordinate origin O position (Shown by the thick dashed line), it can rotate freely within 360 degrees of the xy plane. The optical axis of the optical system is in the z direction shown in FIG. 9. The plane light source ABCD coming from the right side as shown in Figure 9 is focused into a linear light source A'B' through a cylindrical lens. Cylindrical mirror can also be installed on any rotating mechanism to generate line light source A'B' in any direction.

Turn the rotating device shown in Figure 9 to adjust the projection direction of the line light source A'B' on the xy plane, and the angle between A'B' and the x axis is consistent with the rotation angle of the rotating device, which is θ (refer to the figure 10).

FIG. 10 is a schematic diagram of the state of a linear light source with an arbitrary rotation angle and a scanning surface related to the rotating cylindrical mirror.

As shown in Figure 10, the scanning surface abcd as shown in the figure is generated from the line light source A'B'. At this time, both the scanning (reflecting) mirrors SM11 and SM12 of Figure 6 must participate in the scanning, instead of just being shown in Figure 8 Scanning (reflection) mirror SM11 participates in scanning.

Scanning (reflecting) mirrors SM11 and SM12 participate in the scanning technology at the same time. The sawtooth wave shown in Figure 2 is used as the driving signal basis of SM11 and SM12, and then each scanning is performed according to the rotation angle of Figure 10. The (reflective) mirror's due amplitude is multiplied by the respective base signal. As defined in Figure 8 and shown in Figure 10, the amplitude obtained by the scanning base of the scanning (reflection) mirror SM11 is (A'B'/2)sin(θ), and the scanning (reflection) mirror SM12 is obtained by the scanning base The amplitude is (A'B'/2)cos(θ). It should be pointed out that the definition of this scanning direction and rotation direction is arbitrary.

In this case, the relationship (1) can be updated to,

(x _t+1 ,y _t+1 ,θ _t+1 )=(x _t ,y _t ,θ _t )+g(Δx _t ,Δy _t ,Δθ _t ) (2)

Here, θ _t is the angle imposed by the closed-loop control system on the rotating support; (x _t , y _t ) is the translation amount imposed on the scanning (reflection) mirrors SM11 and SM12, and at the same time, (x _t , y _t ) is still The amount of translation superimposed on each of the scanning (reflection) mirrors SM11 and SM12 for generating the scanning signal of FIG. 10. In the same way, in the above formula (2), (x _t , y _t , θ _t ) are the control commands on the mirrors SM11, SM12 and the cylindrical mirror and the line scan camera rotating bracket (equivalent to their respective Movement offset and rotation angle); (Δx _t , Δy _t , Δθ _t ) is the relative movement of the image (target frame) recorded from the line scan camera and the reference image (reference frame); g is the gain of the closed-loop control system ;(X _t+1 ,y _t+1 ,θ _t+1 ) is the next set of new commands applied to the mirrors SM11, SM12, cylindrical mirrors and line scan camera rotation brackets (equivalent to motion Offset and rotation angle).

Figures 6-10 in the above embodiments describe the main LSO imaging system of the present invention, which integrates an internal fundus optical tracking closed-loop control system, such as the control mode of formula (1) or formula (2). On this basis, an auxiliary imaging system as shown in Figure 4 is added. The auxiliary imaging system can be an OCT system, or it can be used for other purposes, such as a fundus laser treatment system. The specific technical implementation of these two parts is described in detail in another patent application.

11 is a schematic diagram of the principle of integrating a main LSO imaging system with closed-loop optical tracking into another auxiliary imaging system according to an embodiment of the present invention.

As shown in Figure 11, the main LSO imaging system on the left integrates another auxiliary imaging system in the upper part, and the main LSO imaging system on the left has its own closed-loop optical tracking function. Its working principle is as follows:

The control signals (x, y, θ) are applied to the scanning (reflection) mirrors SM11, SM12 and the cylindrical mirror L13 of the main LSO system and the rotating bracket of the line scanning camera. The parameters of the control signal are as shown by the dashed line with arrows, which come from the closed-loop control system inside the LSO, and are consistent with formula (1) and formula (2). Compared with the pure digital motion signal of the traditional LSO system, this group of closed-loop control motion signals has the following advantages: 1) smooth; 2) stable; 3) strong anti-interference.

In Figure 11, the control signals (x', y', θ') applied to the scanning (reflection) mirrors SM21 and SM22 of the auxiliary system completely inherit the advantages of the above-mentioned closed-loop control signals (x, y, θ), because (x',y',θ') is obtained by space transformation (x,y,θ), as shown in formula (3):

(x',y',θ')=f(x',y',θ'; x,y,θ)(x,y,θ) (3)

Among them, the spatial transformation relation f (x', y', θ'; x, y, θ) of formula (3) is completely determined by the parameters of the optical system. In formula (3), the spatial transformation relationship f(x',y',θ';x,y,θ) from the main LSO system to the auxiliary system is measured quickly, efficiently, accurately and fully automatically.

The above-mentioned Figures 6-11 describe the optical and mechanical realization of the present invention. The following describes the control implementation part of the embodiment of the present invention, focusing on how to calculate and obtain the fundus position at high speed through algorithms, so as to quickly adjust the scanning (reflection) mirrors SM11, SM12 and the scanning (reflection) mirrors SM21, SM22 to achieve high spatial accuracy and low Time-delayed fundus tracking.

Referring to FIG. 5, the existing data processing technology calculates fundus movement from the LSO image in units of frames. In the embodiment of the invention, frequency doubling technology is used for calculation.

FIG. 12 is a schematic diagram of using frequency doubling technology to reduce the time delay of fundus calculation in an embodiment of the present invention.

As shown in FIG. 12, f ₁ the left side of the image that is consistent with FIG. ₁ and FIG. 5 12A F in the still as a reference frame. The image on the right, that is, f _k in FIG. 12B, is any image of the target frame (k>1). In the present invention, each frame of image is divided into a plurality of equally spaced sub-frame elements, such as S _1,1 , S _1,2 , S _1,3 ,..., according to the time sequence of the data reached by the scanning camera. For the convenience of calculation, S _1,M are all M subframes in the reference frame, _Sk,1 , _Sk,2 , _Sk,3 ,..., _Sk,M are all M subframes in the kth target frame yuan.

Here, the method of the present invention is to scan any frame of image in the scanning direction of SM11 (as mentioned above, usually it is the combination of SM11 and SM12 shown in Figure 10. For convenience, only SM11 shown in Figure 8 is used here as For reference, the following description is the same.) Divided into a plurality of equally spaced subframe elements. The equal spacing means that each sub-frame element contains the same number of scan lines.

Fig. 12A and Fig. 12B show sub-frame elements in the horizontal direction, indicating that the SM11 is scanned in the vertical direction. As shown in Figure 10, the combination of SM11 and SM12 allows the optical system to scan in any direction in a 360-degree space, so the division of sub-frame elements in Figure 12 needs to be adjusted to the corresponding orthogonal direction. For convenience, refer to the SM11 scanning signal in FIG. 2, and refer to FIG. 13 for the division method of the sub-frame elements.

Fig. 13 is a schematic diagram of the scanning (reflecting) mirror SM11 scanning signal and the division of sub-frame elements.

As shown in FIG. 13, the vertical dotted line represents the time (equivalent space) position of each subframe element; the solid thick line represents the sawtooth wave that drives SM11 (or a combination of SM11 and SM12, with the same context) to scan. Normally, sawtooth wave has scan area and return area, as shown in Figure 13. In extreme cases, the time in the return zone is 0, then the sawtooth wave becomes a triangle wave. In the implementation process, a triangular wave can also be used instead of a sawtooth wave as the scanning signal of SM11, as long as the scanning mirrors SM11 and SM12 are not damaged.

In another embodiment of the present invention, a Wasatch Photonics line scan camera (OCTOPLUS3) is used, and the camera receives a 16kHz trigger signal. That is, the camera is set to receive 16,000 line signals per second. In the embodiment, the 16kHz trigger clock is generated from Xilinx FPGA (SP605), or it can be generated from other chips such as DSP.

In the LSO system of the embodiment of the present invention, the SM11 scans each cycle includes 544 lines, of which 512 lines are in the scan area and 32 lines are in the return area. So the frame rate of the image is:

fps=16000/544=29.4

The 512 lines of the scanning area are used for imaging, which is the image shown in Figure 12. The data in the backhaul area is automatically discarded by the system.

The above division method is only one embodiment of the present invention, and different systems may have completely different division methods.

In the case shown in the above embodiment, in this embodiment, a complete scan period of the SM11 is divided into 32 (scan) + 2 (backhaul) sub-areas, and each sub-areas contains 16 scan lines (or time units). As shown by the vertical dashed line in Figure 13, such a complete cycle is exactly 34x16=544 lines.

The key point of the embodiment of the present invention is that once 16 lines arrive at the camera, that is, the data of a sub-frame element is ready, the data of the sub-frame element is immediately sent from the camera to the main PC or other computing units, such as CPU, GPU, DSP, FPGA, etc., the processing unit in the embodiment of the present invention uses nVidia graphics processor GTX1050. The sub-frame metadata of the 16 lines corresponds to one of the positions of _Sk,1 , _Sk,2 , _Sk,3 ,..., _Sk,M in FIG. 12. Obviously, in this example, M=32, which is the total number of frame elements in each frame of image.

Once the calculation unit receives the data of the latest subframe element, the algorithm immediately starts, for example, the Cross Correlation algorithm to calculate the position of the subframe element relative to the reference frame. Normally, it is to find the relative position of the target frame subframe element S _k,m and the reference frame subframe element S _1,m . However, it may also be to find the relative position of the target frame subframe element S _k,m and other reference frame subframe elements S _{1,p (p≠m)} . The above-mentioned specific algorithm implementation process has been disclosed in US Patent No. 9406133.

The advantage of using this method is that the time to obtain a subframe element _Sk,m only requires:

16/16000=1 millisecond;

Instead of waiting for a full frame:

544/16000=34 milliseconds.

After transplanting the Cross Correlation algorithm from the CPU to the nVidia GPU (GTX1050), the mechanical response time from receiving the subframe element _Sk,m data to the motion signal transmission to SM11 and SM12 plus SM11 and SM12 is less than 2 milliseconds. This is equivalent to reducing the total delay time of one cycle of control from (34+2)=36 milliseconds that the best existing device can do to (1+2)=3 milliseconds, the latter being 1/12 of the former .

The best existing device adjusts the frequency of SM11 (without SM12) is the image frame rate of 29.4 Hz, and the device of the present invention adjusts the frequency of SM11 and SM12 to the sampling time of sub-frame element 1000 Hz. This is the frequency multiplication technique mentioned above. Similarly, the specific number here is only an example in the invention. Different systems and different applications can use different parameters to achieve the above frequency multiplication technology.

Compared with the best existing technology, the present invention adopts the technology of transplanting the Cross Correlation algorithm from the CPU to the nVidia GPU (GTX1050), which brings the advantage of improving the spatial accuracy and 3dB time bandwidth of the tracking system by more than an order of magnitude.

Continuing to apply the above example, the data sampling of the frame element of the line scan system can be gradually realized by the following method (refer to Figure 14).

Fig. 14 is a schematic diagram of scanning signals and synchronization signals of a line scanning system.

As shown in Figure 14, the 16kHz line pulse is the system reference clock generated by the FPGA. The scan signal in Figure 13 (the upper part of Figure 14) and the 29.4Hz frame synchronization signal in the lower part of Figure 14 are all obtained from the 16kHz reference pulse phase-locked . In addition, the scan signal and the frame synchronization signal are also completely synchronized. During the ramp of the scan signal, the frame synchronization signal is at a low level; during the backhaul of the scan signal, the frame synchronization signal is at a high level. The generation of these signals can be implemented on FPGA or DSP or other electronic hardware. In the embodiment of the present invention, an FPGA development board SP605 (Spartan 6 chip) of Xilinx is used for implementation.

Normally, controlling the data output mode of the line scan camera is realized by the user inputting a trigger signal of the line scan camera. This trigger signal must include both the 16kHz reference pulse of Figure 14 and the frame synchronization signal of Figure 14, which is a combination of the two. As shown in Figure 15, it is required by the Wasatch Photonics line scan camera OCTOPLUS3 described above. Synchronous trigger signal.

As shown in Figure 15, it shows a signal that combines a line reference clock and a frame synchronization signal to trigger a line scan camera, but the standard method shown in Figure 15 cannot trigger a line scan camera to send a sub-frame of 1000 Hz Metadata. Only using the 16kHz reference clock of Figure 14 can not guarantee the synchronization of the received image and the scan signal. In order to obtain a 1000 Hz sub-frame elementary image synchronized with the scanning signal, the existing trigger technology is also appropriately improved in the embodiment of the present invention.

The trigger signal of the line scan camera only uses the 16kHz reference clock in Figure 14, and the buffer size is 16 lines. This means that the line scan camera regardless of the frame synchronization status, once the line scan camera receives 16 lines of data, it immediately sends it to the PC. However, the embodiment of the present invention makes an additional synchronization in hardware implementation.

Any camera has a state to start and end data sampling. Once the user clicks on the software interface to start sampling, the 16kHz reference clock transmitted to the line scan camera does not start immediately, but waits until the rising or falling edge of the frame synchronization signal to trigger the 16kHz reference clock of the line scan camera. When implementing this function on FPGA in the embodiment of the present invention, the following Verilog code is used:

always@(posedge v_sync)begin

if(camera_start == 1'b0)

camera_trigger<=1'b0;

else

camera_trigger<=line_trigger_16kHz;

end

In the above FPGA code, v_sync is the 29.4Hz frame synchronization signal shown in Figure 14, camera_start is the status register for the user to turn on and off the camera, and camera_trigger is the trigger clock sent to the line scan camera. The code example is the rising edge trigger of v_sync (posedge v_sync), and the other case is the falling edge trigger (negedge v_sync). Only when the rising edge (or falling edge) of _sync and camera_start appear at the same time, the 16kHz reference clock is sent to the line scan camera, otherwise, the line scan camera always gets a low level and is in the sampling waiting state. Sampling here is defined as sending image data from the camera to the receiving device such as PC, GPU, DSP, or other devices.

The trigger difference between rising edge and falling edge is as shown in Figure 14. When the rising edge is triggered, the first and second units of every 34 subframe elements are the data in the return area and need to be eliminated. When the falling edge is triggered, the 33rd and 34th units of every 34 subframe elements are the data in the return area and need to be eliminated.

The specific number described in the above embodiment is only one parameter setting of various embodiments of the present invention, and different systems and different application scenarios can use different parameters. For example, the scanning area can be 1024 lines, and the return area is 32 lines, so the frame rate of the system becomes 16000/(1024+32)=15.2Hz. In addition, according to the parameters of the line scan camera, the frequency of the reference line clock can also be adjusted, from 16kHz to 20kHz or down to 15kHz, etc., which are all parameters that can be changed.

The size of the subframe element can also be adjusted. For example, the above 1000 Hz can be changed to 500 Hz, each sub-frame element has 32 lines. It can also be other sub-frame element sampling frequencies.

The above are only the preferred embodiments of the present invention, and are not used to limit the protection scope of the present invention.

Claims

An image stabilization system based on an improved line scan imaging system, including a line scan fundus camera LSO imaging system, characterized in that the LSO imaging system also includes an orthogonal galvanometer scanning device, used to generate orthogonal in the fundus The scanning surface can be adjusted to any position in the 360-degree space; and used to combine the intelligent control algorithm to realize the LSO optical system to perform two-dimensional fundus scanning while simultaneously performing optical tracking of the fundus inside the LSO.
The image stabilization system based on the improved line scan imaging system according to claim 1, wherein the orthogonal galvanometer scanning device is a double mirror composed of a first mirror (SM11) and a second mirror (SM12). Mirror structure, or MEMS scanning mirror with orthogonal bidirectional vibration, or a structure composed of other orthogonal bidirectional vibration mirrors.
The image stabilization system based on the improved line scan imaging system according to claim 2, wherein the orthogonal galvanometer scanning device combined with an intelligent control algorithm realizes that the LSO optical system performs two-dimensional fundus scanning while simultaneously scanning the fundus inside the LSO For optical tracking, the image stabilization system has the following relationships:

(x t+1 ,y t+1 )=(x t ,y t )+g(Δx t ,Δy t ) (1)

Among them, (x t ,y t ) represents the control commands on the first mirror (SM11) and the second mirror (SM12) at the current sampling time, which are equivalent to their respective motion offsets; (Δx t ,Δy t ) represents the relative motion between the image target frame and the reference frame recorded from the line scan camera; g represents the gain of the closed-loop control system; (x t+1 ,y t+1 ) represents the existing signal applied to the first mirror ( The next set of new commands for SM11) and the second mirror (SM12) are equivalent to the movement offset.
An image stabilization method based on an improved line scan imaging system is characterized in that it comprises the following steps:

A. An orthogonal galvanometer scanning device is added to the online scanning fundus camera LSO imaging system. The orthogonal galvanometer scanning device is used to generate an orthogonal scanning surface on the fundus, so that the scanning surface can be adjusted to any position in a 360-degree space ；

B. Combine the orthogonal galvanometer scanning device with an intelligent control algorithm to realize the LSO optical system to perform the fundus optical tracking inside the LSO while scanning the two-dimensional fundus.
An image stabilization system based on the improved line scan imaging system according to any one of claims 1 to 3, characterized in that it also includes a rotating device for connecting a cylindrical mirror (L13) that generates a line light source and the same The coupled line scan camera is arranged on a 360-degree space controllable rotating bracket, so that the line controllable light source can rotate at any position in the 360-degree space.
The image stabilization system based on the improved line scan imaging system according to claim 5, wherein the sawtooth wave is used as the driving signal base of the first mirror (SM11) and the second mirror (SM12), and the rotation angle is θ, multiplying the due amplitude of each mirror to its respective base signal, then formula (1) can be updated as:

(x t+1 ,y t+1 ,θ t+1 )=(x t ,y t ,θ t )+g(Δx t ,Δy t ,Δθ t ) (2)

Among them, θ t is the angle applied by the closed-loop control system on the rotating bracket; (x t ,y t ) is the translation amount applied on the first mirror (SM11) and the second mirror (SM12), (x t , y t ) or the amount of translation superimposed on each mirror to generate the scan signal; (x t , y t , θ t ) is the current sampling time between the first mirror (SM11) and the second mirror (SM12) and the control commands on the rotating bracket of the cylindrical mirror (L13) and the line scan camera coupled with it, which are equivalent to the respective movement offset and rotation angle; (Δx t , Δy t , Δθ t ) Is the relative motion between the image target frame and the reference frame recorded by the line scan camera; g is the gain of the closed-loop control system; (x t+1 ,y t+1 ,θ t+1 ) is the existing signal applied to the first The next set of new instructions for the mirror (SM11), the second mirror (SM12), and the cylindrical mirror (L13) and the rotation bracket of the line scan camera coupled with it, which is equivalent to the movement offset and rotation angle.
An image stabilization method for an image stabilization system based on an improved line scan imaging system according to claim 5, characterized in that it comprises the following steps:

A. An orthogonal galvanometer scanning device is added to the online scanning fundus camera LSO imaging system. The orthogonal galvanometer scanning device is used to generate an orthogonal scanning surface on the fundus, so that the scanning surface can be adjusted to any position in a 360-degree space ；

B. Mount the cylindrical mirror (L13) that generates the line light source and the line scan camera coupled with it on a 360-degree controllable rotating bracket, so that the line-expanded light source can be rotated at any position in the 360-degree space;

C. Combining the orthogonal galvanometer scanning device with an intelligent control algorithm to realize the LSO optical system to perform the fundus optical tracking inside the LSO while scanning the two-dimensional fundus.