CN114862760B

CN114862760B - Retinopathy of prematurity detection method and device

Info

Publication number: CN114862760B
Application number: CN202210327065.1A
Authority: CN
Inventors: 丁小燕; 谢志; 周昊; 孙立梅; 何尧
Original assignee: Zhongshan Ophthalmic Center
Current assignee: Zhongshan Ophthalmic Center
Priority date: 2022-03-30
Filing date: 2022-03-30
Publication date: 2023-04-28
Anticipated expiration: 2042-03-30
Also published as: CN114862760A

Abstract

The invention discloses a method and a device for detecting retinopathy of prematurity, wherein the method comprises the following steps: s1: acquiring fundus wide-angle images of the left and right sides of an individual to be screened in different directions; s2: sending the fundus wide-angle images with different directions into a pre-trained focus detection model by taking an individual to be screened as a unit to obtain focus detection results of each fundus wide-angle image; s3: the fundus wide-angle images with focus detection results in different directions are spliced and combined to obtain a final spliced image; s4: obtaining fundus retina partitions according to the final spliced image; s5: and visualizing the focus detection result and the partition identification in the partitioned image, and giving a stage result according to the focus type. The invention realizes the complete diagnosis flow of automatically detecting focus, splicing and fusing multi-azimuth images, partitioning, staging and identifying each subtype.

Description

Retinopathy of prematurity detection method and device

Technical Field

The invention relates to the field of image processing, in particular to a method and a device for detecting retinopathy of prematurity.

Background

Retinopathy of prematurity (Retinopathy Of Prematurity, ROP) is a vascular proliferative retinal disease commonly seen in premature infants or very low weight infants, and is the leading cause of blindness in infants. The incidence rate of ROP of infants with birth weight less than or equal to 1500g in China is about 26 percent. Because of the small treatment window period of ROP, early screening and timely intervention are key factors in preventing ROP blinding. The common identification flow of the current fundus screening is as follows: a set of fundus image data of the neonate is acquired using a specialized device, and then the set of images is analyzed and diagnosed by a specialized ophthalmologist to provide a test report.

The difficulty with ROP testing is that the testing procedure from lesion detection analysis to zonal localization of the entire fundus area to disease classification in combination with all information is a cumbersome process and requires years of specialized training to be experienced by an ophthalmic physician to be able to accurately and proficiently give a diagnostic result. The existing methods do not solve all the problems in the whole flow of ROP detection and diagnosis well. The patent (107945870B) realizes the prediction of the ROP lesion stage by establishing a deep neural network model; the patent (108392174B) establishes a set of detection flows of focus detection, key position detection, zoning, stage prediction and additional lesion detection according to the consensus of clinical guidelines; patent (111259982 a) and patent (112308830 a) similarly implement ROP classification and zoning functions, respectively, using an attention convolutional neural network; the patent (111374632A) proposes a method for realizing the detection of key positions and focus by means of quality control, enhancement and other modules; tong Yan et al propose a ROP lesion detection procedure based on a deep convolutional network model to achieve classification of disease based on lesion detection (J.Zhonghua laboratory ophthalmic journal 2019,37 (008): 647-651 and Eye Vis (Lond). 2020Aug 1); the paper (IEEE Trans Med imaging.2021Mar12) designs a novel depth feature fusion network model to predict ROP image classification; james et al use a deep convolutional network to effect the identification of pre-plus, plus disease and normal types (JAMA ophthalmol.2018jul 1).

The main problems of the existing method are: (1) Most of the published work is focused on solving part of the problems in the ROP detection flow, such as plus lesion detection, disease stage and the like; (2) Individual works describe the detection flow entirely according to clinical guidelines, and do not propose a practical method to solve problems in each part, such as partition, critical part detection, etc.; (3) Compared with the traditional diagnosis flow, the method proposed in the prior work lacks the interpretation of the diagnosis result, and the area highly related to the disease is usually represented by the probability of illness or thermodynamic diagram, so that clear diagnosis basis cannot be given; (4) The existing method usually analyzes in a single image mode to give a prediction result, and is different from a single-sample multi-image comprehensive evaluation mode in an actual process, because ROP focus relates to the whole retina area, the existing detection equipment cannot cover the whole area, and therefore, the final diagnosis needs to be evaluated by combining multi-azimuth images, and azimuth information in a plurality of images is mostly lost in the existing method.

Disclosure of Invention

The primary aim of the invention is to provide a method for detecting retinopathy of prematurity, which realizes the complete diagnosis flow of automatic detection of focus, multi-azimuth image splicing and fusion, zoning, stage and subtype identification.

It is a further object of the present invention to provide a retinopathy of prematurity detection device.

In order to solve the technical problems, the technical scheme of the invention is as follows:

a method of retinopathy of prematurity detection comprising the steps of:

s1: acquiring fundus wide-angle images of the left and right sides of an individual to be screened in different directions;

s2: sending the fundus wide-angle images with different directions into a pre-trained focus detection model by taking an individual to be screened as a unit to obtain focus detection results of each fundus wide-angle image;

s3: the fundus wide-angle images with focus detection results in different directions are spliced and combined to obtain a final spliced image;

s4: obtaining fundus retina partitions according to the final spliced image;

s5: and visualizing the focus detection result and the partition identification in the partitioned image, and giving a stage result according to the focus type.

Preferably, in the training of the focus detection model in the step S2, a batch of image data is first labeled, and specific labeling information is a key part in the fundus wide-angle image and a specific type focus, wherein the key part comprises a optic disc and a macula, and the specific type focus comprises but is not limited to vasodilation, neovascularization, hemorrhage, retinal detachment (insufficiency/total), and the like; and obtaining a pre-trained focus detection model by using the marked image data to the focus detection model.

Preferably, in the step S3, the fundus wide-angle images with different orientations of the focus detection result are combined, specifically:

s31: performing feature matching on fundus wide-angle images in different directions to obtain stably matched image pairs a and b;

s32: performing iterative fitting according to a matching point pair set of the stably matched image pairs to obtain a single-mapping transformation matrix H and a matching score from the image b to the image a;

s33: sequentially pairing fundus images in different directions in pairs to obtain all matching scores, sequencing, splicing and fusing the image pair with the highest score, determining whether splicing is successful or not according to the matching score, recording image ids with failed splicing, and storing the image ids in a list L;

s34: repeating the matching and fusing steps of the steps S31 to S33 until all the fundus wide-angle images in the fundus wide-angle Image sequence are processed, obtaining fundus retina images image_1 spliced in different directions, checking a record list L, repeating the steps S31 to S33 once for the images id which are spliced to obtain spliced images image_2, and splicing the images image_1 and image_2 according to the steps S31 to S33 to obtain a final spliced Image.

Preferably, in the step S31, feature matching is performed on fundus wide-angle images in different directions, specifically:

s311: traversing a local feature operator sequence F (F1, F2, …, fn) corresponding to the local key points of the screened image;

s312: calculating feature operators Fa (Fa 1, fa2, …, fan) and Fb (Fb 1, fb2, … fbn) of the image a and the image b respectively, traversing and calculating distance similarity in Fa and Fb in sequence to obtain element pair matching relation S of two sequences, and only reserving nearest distance pairing S-S of each element pair after screening;

s313: dividing an image a and an image b into M grids, and respectively analyzing matching pairs in S-S by taking each grid as a unit, wherein the analysis principle is as follows: if the feature points bi in the matching relation S-S corresponding to the feature points ai in the grid ma in the image a are located in the grid mb in the image b, and at least x matching relation feature points corresponding to the feature points ai in the surrounding neighborhood of the feature points ai in the grid ma are also located in the grid mb, the feature points ai and the feature points bi are considered to be stably matched, the matching relation is reserved, the matching relation which is not matched is removed, and the stable matching relation is obtained after traversing all the grids, so that the stably matched image pairs a and b are obtained.

Preferably, in the step S33, the image pair with the highest score is spliced and fused, specifically:

s331: carrying out affine transformation on the image b by utilizing a single-mapping transformation matrix H to obtain an image b';

s332: respectively calculating the circumscribed circles of the retina areas in the image a and the image b' to obtain circle center and radius information;

s333: the fusion process is performed according to the principle that the edge weights are lower when the fusion process is closer to each other and the center weights of the images are higher, and the fusion process divides the fusion area into three types of intersecting non-boundaries, intersecting boundaries and non-intersecting:

(1) the disjoint areas are filled with corresponding areas by utilizing pixel values corresponding to the image a and the image b' respectively;

(2) intersecting the non-boundary region, calculating distances La and Lb ' of circle centers in the image a and the image b ' corresponding to the current position, and setting pixel filling weights from the image a and the image b ' according to the ratio of the two distances;

(3) and (3) in the intersection boundary region, la and Lb 'are also calculated, and at the moment, the threshold t is introduced to compress or stretch the obtained weights of La and Lb', and the pixel filling is completed according to the final weights.

Preferably, after the final stitched image is obtained in step S3, coordinate mapping is further performed on focus information in the image sequence according to the stitched position, specifically:

and respectively storing position coordinates (x, y) according to the categories of different focuses or key position points, respectively calculating a single-map transformation matrix H of each image obtained by matching with the position coordinates to obtain fused position coordinates (x ', y') = (x, y) H, and combining focus information by utilizing the same type and similar coordinate region combining principle to determine the positions of the video disc and the macula lutea in the key parts of the subarea.

Preferably, after the positions of the optic disc and the macula in the key part of the subarea are determined, the positions of the sawtooth rims are determined by the positions Loc of the optic disc in the pre-designed fundus-based wide-angle image _{Vision device} Macula position Loc _{Yellow colour} Wide angle image width I _{Wide width of} The mapping relation F of (2) is calculated, and the formula is as follows:

Loc _saw ＝F(Loc _{Vision device} ,Loc _{Yellow colour} ,I _{Wide width of} )，

In Loc _Saw Is the location of the serrated edge.

Preferably, the determining of the mapping relation F specifically includes:

preparing a plurality of groups of wide-angle fundus images of different directions obtained by the same individual { [ a1, a2, …, an ]]，[b1,b2,…,bn]… and a complete retinal fundus wide area image containing a serrated edge [ A, B, … ]]Establishing a position mapping relation from a wide-angle image set to a wide-angle image, matching the wide-angle image sets a1, a2 and … an with the image A respectively to obtain corresponding single-mapping transformation matrixes h1, h2, … and hn, matching the images in the sets to corresponding positions in the image A, judging that saw-tooth edge position point coordinate information Loc saw appears in the wide-angle image through the positions of key positions of saw-tooth edges marked in advance in the image A, and enabling Loc to be realized _Saw ＝F(Loc _{Vision device} ,Loc _{Yellow colour} ,I _{Wide width of} ) And (3) taking the plurality of groups of data as training data, training a fitting polynomial F to obtain a mapping function F of the positions of the video disc, the macula lutea and the image on the wide-angle image from wide to the sawtooth edge, and then directly using a mapping function F model to carry out the position estimation of the sawtooth edge.

Preferably, in the step S4, a fundus retina partition is obtained according to the final stitched image, specifically:

taking the position of the video disc as the center of a circle, taking twice the distance from the video disc to the macula as the radius, wherein the circular area is an area I; taking the optic disc as the center of a circle, taking the saw-tooth edge of the optic disc as the radius to make a circle, taking the circular ring area after removing the area I as the area II, and taking the whole fundus area as the area III except the area I and the area II.

A retinopathy of prematurity detection device comprising:

the image acquisition module is used for acquiring fundus wide-angle images of the left side and the right side of the individual to be screened in different directions;

the detection module sends fundus wide-angle images with different directions into a focus detection model trained in advance by taking an individual to be screened as a unit to obtain focus detection results of each fundus wide-angle image;

the image merging module is used for merging the fundus wide-angle images with focus detection results in different directions to obtain a final merged image;

the partitioning module is used for obtaining fundus retina partitions according to the final spliced image;

and the analysis module visualizes the focus detection result and the partition identification in the partitioned image and gives a stage result according to the focus type.

Compared with the prior art, the technical scheme of the invention has the beneficial effects that:

(1) The invention realizes the complete diagnosis flow of automatically detecting focus, splicing and fusing multi-azimuth images, partitioning, staging and identifying each subtype;

(2) According to the invention, through integrating multi-azimuth image information, focus and subarea in a complete fundus retina area are intuitively displayed, statistics information of clock count is displayed, and a doctor can select to directly use an analysis result or autonomously give a diagnosis result according to a visual basis;

(3) The invention provides an objective and feasible method for automatically identifying the key parts (sawtooth edges) of the fundus by modeling by using a wide-angle image and a wide-area image;

(4) Compared with the method of sequentially inputting and sequentially outputting the predicted result in the prior work, the method of the invention is more in line with the conventional diagnosis and analysis flow.

Drawings

FIG. 1 is a schematic flow chart of the method of the present invention.

Fig. 2 is a flow chart of image merging according to the present invention.

Fig. 3 is a flow chart of image feature matching according to the present invention.

Fig. 4 is a schematic flow chart of the highest scoring image pair stitching fusion of the present invention.

Fig. 5 is a schematic diagram of an image stitching fusion result provided in the embodiment.

Fig. 6 is a schematic diagram of a focus information fusion result provided in the example.

Fig. 7 is a schematic diagram of a wide area image sawtooth edge labeling provided in an embodiment.

Fig. 8 is a schematic diagram of the wide-angle image map of fig. 7 to a wide-area image.

Fig. 9 is a schematic diagram of lesion merging and zoning identification provided in the examples.

Fig. 10 is a schematic view of the device module of the present invention.

Detailed Description

The drawings are for illustrative purposes only and are not to be construed as limiting the present patent;

for the purpose of better illustrating the embodiments, certain elements of the drawings may be omitted, enlarged or reduced and do not represent the actual product dimensions;

it will be appreciated by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.

The technical scheme of the invention is further described below with reference to the accompanying drawings and examples.

Example 1

The present embodiment provides a method for detecting retinopathy of prematurity, as shown in fig. 1, comprising the steps of:

s4: obtaining fundus retina partitions according to the final spliced image;

In a specific implementation process, the embodiment provides a clinical automatic auxiliary diagnosis process comprising functions of focus detection, fundus partition positioning, multi-azimuth image combination, ROP stage analysis and the like. The embodiment can respectively detect the focus type and the position of the multi-azimuth fundus image acquired by each case, and then combine and splice the images in different azimuth, as shown in fig. 5; simultaneously merging the same focus in the images of different directions, as shown in fig. 6; obtaining the position of the partitioned positioning key part by designing and defining a mapping relation model between a wide angle and a wide area fundus image to complete the regional division, as shown in fig. 7 and 8; the relationship between the comprehensive area and the focus determines the stage of the disease and the judgment of various additional lesions, and visualizes the detection result to an operator, as shown in fig. 9, the operator can choose to directly adopt the analysis result or adjust the model analysis result to finally give a diagnosis report.

Example 2

The present embodiment provides the training procedure of the focus detection model trained in advance in step S2 on the basis of embodiment 1:

firstly, marking a batch of image data for training a focus detection model, wherein specific marking information is a key part and a specific type focus in a fundus wide-angle image, and the key part comprises a video disc and a macula lutea; and obtaining a pre-trained focus detection model by using the marked image data to the focus detection model.

Example 3

In this embodiment, based on embodiment 1, a specific flow is provided for merging fundus wide-angle images with different orientations of the focus detection result in step S3, as shown in fig. 2, specifically:

In step S31, feature matching is performed on fundus wide-angle images in different directions, as shown in fig. 3, specifically:

The step S32 obtains a single-mapping transformation matrix H and a matching score from the image b to the image a, specifically:

s321: filling the boundary of the image a and the image b;

s322: separating image channels and extracting G channels;

s323: segmenting a fundus image region ROI;

s324: corroding the ROI to obtain a mask;

s325: traversing to detect and calculate local feature vectors in the mask;

s326: calculating nearest neighbor distances according to the local feature vectors of the image a and the image b in pairs;

s327: based on the feature matching of grid statistics, eliminating error matching;

s328: judging whether the matching score is required to be calculated, if so, evaluating the matching score according to the quality and the quantity of the matched feature vector pair, and if not, entering step S329;

s329: further eliminating error matching based on a random sampling consistency algorithm, reserving main matching pairs, and fitting to obtain a single mapping matrix;

s3210: obtaining a single-mapping transformation matrix H from the image b to the image a according to the matching pair, and evaluating scale, angle and displacement transformation indexes in the affine transformation matrix;

s3211: judging whether the index is suitable, if so, obtaining a single-mapping transformation matrix H from the image b to the image a, and if not, failing to match.

In the step S33, the image pair with the highest score is spliced and fused, as shown in fig. 4, specifically:

Example 4

In this embodiment, on the basis of embodiment 1, after the final stitched image obtained in step S3 is continuously disclosed, coordinate mapping is further performed on focus information in the image sequence according to the stitched position, which specifically includes:

After the positions of the optic disc and the macula in the key part of the subarea are determined, the positions of the sawtooth edge are determined by the positions Loc of the optic disc in the pre-designed fundus-based wide-angle image _{Vision device} Macula position Loc _{Yellow colour} Wide angle image width I _{Wide width of} The mapping relation F of (2) is calculated, and the formula is as follows:

In Loc _Saw Is the location of the serrated edge.

The determination of the mapping relation F specifically comprises the following steps:

preparing a plurality of groups of wide-angle fundus images of different directions obtained by the same individual { [ a1, a2, …, an ]]，[b1,b2,…,bn]… and a complete retinal fundus wide area image containing a serrated edge [ A, B, … ]]Establishing a position mapping relation from a wide-angle image set to a wide-angle image, respectively matching the wide-angle image sets a1, a2 and … an with the image A to obtain corresponding single-mapping transformation matrixes h1, h2, … and hn, matching the images in the sets to corresponding positions in the image A, and passing through saw teeth marked in advance in the image AThe position of the key position of the edge can judge the coordinate information Loc saw of the position point of the sawtooth edge in the wide-angle diagram, so that the Loc _Saw ＝F(Loc _{Vision device} ,Loc _{Yellow colour} ,I _{Wide width of} ) And (3) taking the plurality of groups of data as training data, training a fitting polynomial F to obtain a mapping function F of the positions of the video disc, the macula lutea and the image on the wide-angle image from wide to the sawtooth edge, and then directly using a mapping function F model to carry out the position estimation of the sawtooth edge.

Example 5

The present embodiment further discloses, based on embodiment 1, that in step S4, the fundus retina partition is obtained according to the final stitched image, specifically:

Visualizing the focus detection result and the partition mark in the partitioned image, and giving a stage result according to the focus type; determining various ROP subtypes according to the partition information corresponding to the focus types and the positions; according to the regional distribution condition of the focus, the distribution Zhong Dianshu of a focus of a certain type is counted, and the information is displayed in a structured way after being integrated.

Example 6

The present embodiment provides a retinopathy of prematurity detection apparatus, as shown in fig. 10, comprising:

The same or similar reference numerals correspond to the same or similar components;

the terms describing the positional relationship in the drawings are merely illustrative, and are not to be construed as limiting the present patent;

it is to be understood that the above examples of the present invention are provided by way of illustration only and not by way of limitation of the embodiments of the present invention. Other variations or modifications of the above teachings will be apparent to those of ordinary skill in the art. It is not necessary here nor is it exhaustive of all embodiments. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the invention are desired to be protected by the following claims.

Claims

1. A method for detecting retinopathy of prematurity comprising the steps of:

s4: obtaining fundus retina partitions according to the final spliced image;

s5: visualizing the focus detection result and the partition mark in the partitioned image, and giving a stage result according to the focus type;

in the step S3, the fundus wide-angle images with focus detection results in different directions are combined, specifically:

s34: repeating the matching and fusing steps from S31 to S33 until all fundus wide-angle images in the fundus wide-angle Image sequence are processed, obtaining fundus retina images image_1 spliced in different directions at the moment, checking a record list L at the moment, repeating the steps from S31 to S33 once for the images id which are spliced to obtain spliced images image_2, and splicing the images image_1 and image_2 according to the steps from S31 to S33 to obtain a final spliced Image;

in the step S33, the image pair with the highest score is spliced and fused, specifically:

2. The method according to claim 1, wherein the training of the lesion detection model in step S2 requires labeling a batch of image data, and the specific labeling information is a critical part and a specific type of lesion in the fundus wide-angle image, wherein the critical part includes optic disc and macula; and obtaining a pre-trained focus detection model by using the marked image data to the focus detection model.

3. The method according to claim 1, wherein the feature matching is performed on fundus wide-angle images of different orientations in step S31, specifically:

4. The method of claim 3, wherein after obtaining the final stitched image in step S3, the coordinate mapping is further performed on the lesion information in the image sequence according to the stitched position, specifically:

5. The method according to claim 4, wherein the positions of the sawtooth rims after determining the positions of the optic disc and the macula in the key parts of the division are determined from the positions Loc of the optic disc in the fundus wide-angle image based on the design in advance _{Vision device} Macula position Loc _{Yellow colour} Wide angle image width I _{Wide width of} The mapping relation F of (2) is calculated, and the formula is as follows:

In Loc _Saw Is the location of the serrated edge.

6. The method for detecting retinopathy of prematurity according to claim 5, wherein the determining of the mapping relation F specifically comprises:

7. The method according to claim 6, wherein the step S4 is performed to obtain the fundus retina partition from the final stitched image, specifically:

8. A retinopathy of prematurity detection device comprising:

the image acquisition module is used for acquiring fundus wide-angle images of different directions on the left side and the right side of an individual to be screened;

the analysis module visualizes the focus detection result and the partition mark in the partitioned image and gives a stage result according to the focus type;

the image merging module is used for merging fundus wide-angle images with focus detection results in different directions, and specifically comprises the following steps:

performing feature matching on fundus wide-angle images in different directions to obtain stably matched image pairs a and b;

performing iterative fitting according to a matching point pair set of the stably matched image pairs to obtain a single-mapping transformation matrix H and a matching score from the image b to the image a;

sequentially pairing fundus images in different directions in pairs to obtain all matching scores, sequencing, splicing and fusing the image pair with the highest score, determining whether splicing is successful or not according to the matching score, recording image ids with failed splicing, and storing the image ids in a list L;

repeating the matching and fusing steps until all fundus wide-angle images in the fundus wide-angle Image sequence are processed, obtaining fundus retina images image_1 spliced in different directions at the moment, checking a record list L at the moment, repeating the steps S31 to S33 for one time for the images id which are failed to be spliced to obtain spliced images image_2, and splicing the images_1 and image_2 according to the steps S31 to S33 to obtain final spliced images;

the image pair with the highest score is spliced and fused, specifically:

carrying out affine transformation on the image b by utilizing a single-mapping transformation matrix H to obtain an image b';

respectively calculating the circumscribed circles of the retina areas in the image a and the image b' to obtain circle center and radius information;

the fusion process is performed according to the principle that the edge weights are lower when the fusion process is closer to each other and the center weights of the images are higher, and the fusion process divides the fusion area into three types of intersecting non-boundaries, intersecting boundaries and non-intersecting: