CN115047610B

CN115047610B - Chromosome karyotype analysis device and method for automatically fitting microscopic focusing plane

Info

Publication number: CN115047610B
Application number: CN202210984995.4A
Authority: CN
Inventors: 韦然; 宋宁; 晏青; 吕明; 马伟旗; 唐悦
Original assignee: Hangzhou Daigens Biotech Ltd
Current assignee: Hangzhou Daigens Biotech Ltd
Priority date: 2022-08-17
Filing date: 2022-08-17
Publication date: 2022-12-27
Anticipated expiration: 2042-08-17
Also published as: CN115047610A

Abstract

The invention provides a chromosome karyotype analysis device and method for automatically fitting a microscopic focusing plane, wherein the device comprises a panoramic camera carrying a narrow-band filter with a specific waveband, a microscope provided with 10-fold and 100-fold objective lenses, a high signal-to-noise ratio industrial camera, a motor and an intelligent control center, a colony coordinate in a visual field is obtained on a clearest image through a colony detection network by obtaining an initial global image of a slide, a z value corresponding to a maximum value on each pre-scanning point in a definition fractional space is taken, space fitting is carried out to obtain a final z value focal plane, the optimal effect of the full-slide focal plane fitting is obtained on the premise of not increasing time consumption, accurate focusing of subsequent microscopic shooting can be guided, and efficient shooting is realized.

Description

Chromosome karyotype analysis device and method for automatically fitting microscopic focusing plane

Technical Field

The invention relates to the technical field of chromosome karyotype analysis, in particular to a chromosome karyotype analysis device and method capable of automatically fitting a microscopic focusing plane.

Background

Photomicrographs have been widely used in histopathology, cytology, and microscopic morphology observation and study of genetic chromosomes. With the continuous development of scientific technology, the degree of computer analysis has appeared, but most of chromosome examinations still rely on the traditional method, and the success or failure of photomicrography directly affects the correct analysis of chromosomes.

First, the prior art device simply selects 25 pre-scan points of the coverslip globally using the slide when fitting the focal plane before photomicrography of the chromosome colony, as shown in fig. 1, which leads to two problems: (1) focusing failure or large focusing error is caused by the prospect of non-nucleated communities and the like in the field of view of the pre-scanning point; (2) a full slide pre-scan can consume more time.

Secondly, when focusing is performed on scanning points, the conventional equipment usually adopts a general focusing method, but a slide used for chromosome karyotype diagnosis usually has a large amount of impurity backgrounds, as shown in fig. 2, if direct focusing is performed, the clearest z value (longitudinal coordinate) corresponding to a karyotype community of the scanning point may not be obtained, and the error of each pre-scanning point will be accumulated to the subsequent focusing plane fitting, so that the shooting effect of community whole slide scanning under a subsequent 10-time objective lens is significantly influenced.

Thirdly, even when the focusing plane fitting is performed after the clearest z value corresponding to the scanning point karyotype community is obtained, the existing device often adopts a simple interpolation method of the optimal z value point, which may cause the following problems: (1) each pre-scanning point adopts a single optimal z value point, so that the error of the single z value point in the process of fitting the focal plane is enlarged to the whole influence.

Disclosure of Invention

In order to overcome the defects in the prior art, the invention provides a chromosome karyotype analysis device and method for automatically fitting a microscopic focusing plane.

In order to achieve the above object, the present invention provides a chromosome karyotype analysis apparatus for automatically fitting a microscopic focusing plane, comprising a microscope and a motor both electrically connected to an analysis control center, wherein the analysis control center is used for data receiving and processing, the microscope is provided with a stage and an objective lens, the objective lens is provided with a camera, a slide to be analyzed is arranged on the stage, the motor is connected to the stage, and there is no relative displacement in the vertical direction between the motor and the slide to be analyzed, and the motor controls the relative movement between the objective lens and the slide to be analyzed; (ii) a The motor adjusts the movement of the slide to be analyzed in the horizontal direction through the push-pull piece.

The invention provides a method for automatically fitting a chromosome karyotype analysis device for automatically fitting a microscopic focal plane, which comprises the following steps:

acquiring an initial global image of the slide, and acquiring an inscribed rectangle with the largest distribution area of the coverage karyotype community according to the distribution condition of the chromosome karyotype community;

setting a plurality of pre-scanning points which are uniformly distributed on the obtained inscribed rectangle with the largest area, carrying out focusing change shooting on the pre-scanning points one by one under a 10-time mirror to obtain a plurality of images, and determining the clearest image and a z value corresponding to the longitudinal coordinate of the objective table by utilizing a definition evaluating algorithm;

obtaining community coordinates in a visual field on the clearest image through a community detection network, focusing again by taking the community coordinates as a focusing area, reducing the focusing variation degree, photographing again and evaluating the definition to obtain definition scores and corresponding z values of a plurality of chromosome images;

selecting the first n (n is more than or equal to 2) clearest chromosome images, and performing spatial coordinate interpolation on all pre-scanning points by adopting definition values and corresponding z values to obtain a definition score space with the thickness of n and a z value space with the thickness of n;

taking a z value corresponding to the maximum value on each pre-scanning point in the definition fractional space, and performing space fitting to obtain a final z value to a focal plane;

the community detection network performs feature extraction by taking a lightweight backbone network mobilene _ v3 as a feature extraction layer to obtain output feature maps corresponding to three levels close to the output layer in the backbone network, wherein the feature extraction layer comprises two modes of feature fusion from bottom to top and feature fusion from top to bottom, so that recursive fusion of the feature maps is realized.

Preferably, the community detection network performs feature extraction by taking a lightweight backbone network mobilene _ v3 as a feature extraction layer to obtain output feature maps corresponding to three levels close to the output layer in the backbone network, and the feature extraction layer includes two modes of feature fusion from bottom to top and feature fusion from top to bottom, so as to realize recursive fusion of the feature maps.

Preferably, the bottom-up feature fusion enlarges the small feature map of the lower layer to the same size as the feature map of the previous level in an upsampling mode, so as to realize the fusion of the stronger semantic features of the lower layer into the feature map of the upper layer.

Preferably, the top-down feature fusion is performed by means of downsampling, and the feature map of each stage is downsampled and then sent back to the next-layer feature map.

Preferably, after feature extraction and feature fusion, the community detection network directly predicts the corresponding detection frame through convolution in a yolov5 mode on the target frame prediction layer, and then automatically amplifies and inputs the local area to be amplified.

Preferably, the automatic amplification input is to uniformly divide the image, match each detection frame with the region where the detection frame is located, acquire the minimum circumscribed rectangle and the direction of each detection frame, and expand the minimum circumscribed rectangle to the image boundary according to the corresponding direction, so as to obtain the local region to be amplified; intercepting the original input corresponding image part, amplifying in proportion, performing feature extraction and recursive fusion again through a community detection network, and mapping the corresponding target frame back to the original input image after the target frame of the corresponding local area is obtained at the target frame prediction layer.

Preferably, the spatial fitting is to determine the z value when all the pre-scanning points are clearest, and then perform interpolation according to the whole plane to obtain the corresponding z values of other points except all the pre-scanning points on the plane, so as to obtain the focal plane completely covering all the pre-scanning areas.

Preferably, the number of the pre-scanning points is 25.

The invention has the following beneficial effects:

1. the method has the advantages that the panoramic camera carrying the narrow-band filter with the specific wave band is additionally arranged to obtain the global view of the slide, so that the dense distribution condition of the chromosome colony is obtained in advance, the effective dense distribution range of the chromosome colony is selected for laying the pre-scanning points, the condition that the pre-scanning cannot be effectively focused due to less background during focusing can be effectively avoided, and meanwhile, the scanning time of the subsequent 10 times of the objective lens full slide can be reduced by reducing the scanning range;

2. a community coordinate in a visual field is obtained through a lightweight community detection network, a focusing area is determined according to the community coordinate, and definition evaluation is performed, so that a definition evaluation result of a background of a chromosome community position is obtained, and two benefits can be brought: (1) focusing is carried out by the chromosome community, each pre-scanning point can obtain more accurate pre-scanning focusing precision, so that a focusing target is directly associated with a shooting target, and the influence of redundant background and the like is reduced; (2) because the influence of the background is removed, the measurement objects are unified, so that the condition that the reference z value and the definition can be directly compared and scored is obtained between each pre-scanning point, and the focusing plane established based on each pre-scanning point is more accurate;

3. by integrating the z value and the definition for scoring, the optimal effect of the focal plane fitting of the whole slide is obtained on the premise of not increasing the time consumption, the accurate focusing of the subsequent microscopic shooting can be guided, and the efficient shooting is realized.

Drawings

FIG. 1 is a schematic diagram of a prior art apparatus for setting up 25 pre-scan points globally across a slide;

FIG. 2 is a diagram showing the distribution of chromosome communities and impurities in a visual field when microscopic photography is performed by the existing equipment, wherein discrete origins are impurities, and clustered small spots are medium chromosome communities;

FIG. 3 is a schematic diagram showing the arrangement of pre-scanning points in an effective dense distribution range of chromosome communities according to the present invention; FIG. 4 is a schematic diagram of determining a focusing area by obtaining a community coordinate in a field of view through a lightweight community detection network according to the present invention;

FIG. 5 is a schematic diagram of an inscribed rectangle for obtaining distribution of a covered karyotype community according to specific distribution of the slide karyotype community, wherein a diagram A is an image of a panoramic camera carrying a narrow-band filter with a specific wave band for photographing a slide, a diagram B is a state of the image after sharpening pretreatment, and diagrams C and D are front and back state diagrams of a maximum inscribed rectangle for obtaining distribution of the covered karyotype community;

FIG. 6 is a schematic diagram of a community detection network;

FIG. 7 is a schematic diagram of a plan for four "local regions" for magnification;

FIG. 8 is an exemplary diagram of focal plane fitting in the present invention;

FIG. 9 is a diagram showing the results of a fit to the focal plane of a pre-scan region;

FIG. 10 is a view showing an overall configuration of an apparatus for analyzing a karyotype based on an auto-fitting microscopic focal plane according to the present invention;

FIG. 11 is a logic diagram of a sharpness evaluation algorithm according to the present invention.

Detailed Description

To better illustrate the objects, aspects and advantages of the present invention, the present application will be further described with reference to specific examples.

Example 1:

the invention provides a chromosome karyotype analysis device capable of automatically fitting a microscopic focusing plane, as shown in fig. 10, the device comprises a microscope 200 and a motor 300 which are both electrically connected to an analysis control center 100, the analysis control center 100 is used for image receiving, definition analysis of a microscopic shooting diagnosis instrument and data receiving and processing of a vertical coordinate of the motor 300, for example, a Metasight ([ methyl ]) full-automatic cell microscopic image scanning system, the microscope 200 is provided with a carrying platform 201 and an objective lens 202, a slide to be analyzed is arranged on the carrying platform 201, a camera is arranged on the objective lens 202 to realize shooting of the slide to be analyzed, the motor 300 is connected with the carrying platform 201, the motor 300 and the slide to be analyzed do not have relative displacement in the vertical direction, the motor 300 controls the vertical movement of the carrying platform 201 to realize adjustment of the relative displacement between the objective lens 202 and the slide to be analyzed, preferably, the motor 300 selects a stepping motor, and the specification of the slide to be analyzed is 75mm × 25mm × 1.1mm. The motor 300 is further connected to a push-pull member, and the horizontal movement of the slide to be analyzed is controlled by the push-pull member, which is set in CN 112859316A.

The method of use of the apparatus can be summarized as the following process:

determining chromosome community distribution condition and laying pre-scanning points in a targeted manner

Carrying a narrow-band filter with a specific waveband through a panoramic camera to photograph the slide so as to obtain an initial panoramic picture of the slide; highlighting the specific distribution of the karyotype community through image preprocessing methods such as sharpening and the like, and obtaining an inscribed rectangle with the largest area for covering the distribution of the karyotype community according to the distribution condition;

25 pre-scanning points of 5x5 lines and rows which are uniformly distributed are arranged on the obtained inscribed rectangle with the largest area, continuous coarse grain (0.02 micrometer) z value variation shooting within the range of 0.4 micrometer of the lower stroke of a low-power mirror (10-power mirror) is carried out on the pre-scanning points one by one, and the clearest z value and the corresponding image are obtained through a definition evaluation algorithm;

obtaining the coordinate of the colony in the visual field through a colony detection network (as shown in fig. 6) on a clear image, directly determining a 'focusing area' by using the colony coordinate, and acquiring the vicinity of the z value in the first stage on the basis of the 'focusing area' and the 'focusing area' (the vicinity of the z value is obtained by using the colony coordinate: (shown in fig. 6)

0.05 micron) to perform further fine-grained (0.005 micron) z value variable definition evaluation, thereby obtaining definition scores of multiple chromosome images on definition thresholds and corresponding z values.

Referring to fig. 11, the above sharpness evaluation algorithm can be summarized as the following process: the method comprises the steps of performing Gaussian blur processing on an image to be evaluated once to obtain a degraded image of the image, then comparing the change conditions of adjacent pixel values of an original image and the degraded image, determining the definition value according to the change conditions, wherein the smaller the calculation result is, the clearer the image is, and the clearer the image is, otherwise, the more fuzzy the image is.

Improved community detection network for improving chromosome community detection precision

The resolution of the input image for colony detection is relatively large (1600 x 1100), however, the detected chromosome colony is often relatively small (about 80x 80), and the original image is directly input into the detection network, which often has a relatively high rate of missed detection and false detection due to relatively small targets, and this may significantly affect the above-mentioned effect of determining a "focusing area" for focusing based on the colony detection result. On one hand, chromosome community detection needs to involve chromosome community detection under various medical scenes such as peripheral blood, amniotic fluid and bone marrow, wherein the number of chromosome communities contained in peripheral blood and amniotic fluid samples is large, and certain tolerance is provided for community omission detection, but the number of chromosome communities in bone marrow samples is small, so that the community detection sensitivity is required to be high. On the other hand, the background of the image where the chromosome community is located is complex, and a large number of impurities and the chromosome community often have high similarity under a large visual field, so that if the false detection rate of the community detection network is too high, the determination of the 'focusing area' based on the community detection result is greatly interfered by background noise. Therefore, in order to improve the accuracy of chromosome community detection, the invention is mainly improved from two aspects:

increasing recursive feature fusion, and improving the detection capability of small targets:

the community detection network obtains output characteristic diagrams corresponding to three levels (stages) close to an output layer in the backbone network by taking the lightweight backbone network mobilene _ v3 as a characteristic extraction layer. In the bottom-up feature fusion process, the small feature map of the lower layer is amplified to the same size as the feature map of the previous stage in an up-sampling mode, so that stronger semantic features (beneficial to classification) of the lower layer are fused into the feature map of the upper layer, the semantic expression on the feature map of each stage is enhanced, and the up-sampling method can be realized by using nearest neighbor difference values. Furthermore, the invention sends the feature map of each stage back to the next layer of feature map after down sampling, thereby realizing the recursive fusion of the feature maps, and the benefit of doing so is that the high resolution information (beneficial to positioning) of the upper layer is fused into the feature map of the lower layer, thereby enhancing the positioning information on the feature map of each stage, and simultaneously reducing the number of model parameters compared with the additional addition of a top-down feature fusion mode, thereby accelerating the detection speed of the community.

II, a two-stage community detection network based on a local automatic amplification mechanism:

after the feature extraction and the feature fusion, the mode of yolov5 is adopted on a target frame prediction layer to directly predict a corresponding detection frame through convolution.

After all the initial target frames are obtained, the image is divided into four regions according to the center of the image, then each initial target frame is distributed to the corresponding region, and meanwhile, the distribution rule of the initial target frames in the multiple regions is the region distributed with the area ratio of the region. After the "initial target frames" corresponding to the four regions are obtained, the minimum bounding rectangle and the direction thereof are obtained according to each target frame, and then 100 pixels are expanded (or expanded to the boundary) in four directions, namely, the upper direction, the lower direction, the left direction and the right direction, so as to obtain four to-be-amplified "local regions".

After four local areas are obtained, the original input corresponding image part is intercepted, then the proportion is kept (the proportion corresponding to 16x11 is obtained by adding a border), then the proportion is amplified to 1600x1100, the image is input into the feature extraction and fusion module again, after a target frame corresponding to the local area is obtained through a target frame prediction layer, the corresponding target frame is mapped back to the original input image, and at the same time, the final detection result is directly output after the judgment of the counting judgment logic is negative.

Fitting of spatial focal planes

Determining the clearest number n (taking 3 by default) in the obtained z values and the definition scores corresponding to different clear images of the number of the images under 25 pre-scanning points, then obtaining a definition score space with the thickness of n and a z value space with the thickness of n according to the spatial coordinate interpolation of the 25 points, and taking a z value corresponding to the maximum value on each coordinate point in the definition score space with the thickness of n for spatial fitting, thereby obtaining the final z value to the focal plane.

The actual image definition evaluating algorithm is used for measuring the 'unblurry degree' of an image, the image definition has a relatively accurate standard, but the image is difficult to accurately describe due to the fact that the image is not clear, the actual image definition evaluating algorithm is embodied as a clearer image, the absolute value and the relative value of the definition are always relatively large in reference meaning, and the absolute value and the relative value of the definition of the clearer image are relatively small in reference meaning.

The focal plane fitting mode can be summarized as that a z value of the clearest point is obtained by focusing and performing definition evaluation through 25 pre-scanning, and then the z value is taken as a representative of the point, so that interpolation is performed according to the whole plane to obtain the corresponding z values of other points except the 25 pre-scanning points on the plane.

As shown in fig. 8, specifically, there are four pre-scanning points a, B, C, and D, each coordinate point, for example, a (0.205,95), takes the z value and the corresponding sharpness score corresponding to the clearest three pictures, now a coordinate point E is given, falls into the ABCD area, and the abscissa and ordinate ratios from the four points ABCD are shown as follows, in order to find the z value corresponding to the E point:

a simpler method is to directly take the clearest z value of the ABCD three points to carry out interpolation so as to directly obtain the z value of the E point, and the calculation process is as follows:

(1) Firstly, calculating a temporary point X parallel to the point E in the AD point connecting line, and calculating to obtain a z value as follows:

(2) Then calculating a temporary point Y parallel to the point E in the BC point connecting line, and calculating to obtain a z value as follows:

(3) Then calculating according to the X point and the Y point to obtain the z value of the E point as follows:

by the pixel-by-pixel traversal interpolation method, a focal plane completely covering 25 pre-scanning areas can be obtained, and the final effect is shown in fig. 9.

The method only takes the threshold value of one clearest image of each pre-scanning point, is easily interfered by a definition evaluation algorithm to random errors of a single picture, and does not fully utilize useful information in the process of focusing each pre-scanning point for multiple times, so that the error between a z-value point obtained by final interpolation and an actual z-value point is larger. Therefore, the invention carries out comprehensive interpolation on each pre-scanning point by taking the z value and the definition score of the clearest sheet number n (3 sheets are taken as a default), and the specific calculation process is as follows:

(1) Firstly, calculating a temporary point X parallel to the point E in the AD point connecting line, and calculating to obtain a z value and a definition score corresponding to the clearest 3 graphs:

(2) Then calculating a temporary point Y parallel to the point E in the BC point connecting line, and calculating to obtain a z value and a definition score corresponding to the clearest 3 graphs:

(3) Then calculating the z value and the definition score of the clearest 3 graphs of the E point according to the X point and the Y point:

and taking the z-value point with the highest resolution score, namely 0.205625 corresponding to 93.500 as the final z value of the E point.

Finally, it should be noted that the above embodiments are only used for illustrating the technical solutions of the present invention and not for limiting the protection scope of the present invention, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions can be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for using a karyotype analysis apparatus for automatically fitting a microscopic focal plane, comprising: the microscope and the motor are both electrically connected to an analysis control center, the analysis control center is used for receiving and processing data, the microscope is provided with an objective table and an objective lens, the objective lens is provided with a camera, a slide to be analyzed is arranged on the objective table, the motor is connected with the objective table, no relative displacement exists between the motor and the slide to be analyzed in the vertical direction, and the motor is used for controlling the relative movement between the objective lens and the slide to be analyzed; the motor adjusts the movement of the slide to be analyzed in the horizontal direction through the push-pull piece;

the use method of the device comprises the following steps:

acquiring an initial global image of the slide, and acquiring an inscribed rectangle with the largest area for covering the distribution of the karyotype community according to the distribution condition of the chromosome karyotype community;

selecting the first n clearest chromosome images, wherein n is more than or equal to 2, and performing spatial coordinate interpolation on all pre-scanning points by adopting definition values and corresponding z values to obtain a definition score space with the thickness of n and a z value space with the thickness of n;

2. The method for using the apparatus for analyzing karyotype with automatic fitting of microscopic focusing plane according to claim 1, wherein: the bottom-up feature fusion enlarges the small feature map of the lower layer to the same size as the feature map of the previous layer in an upsampling mode.

3. The method of using the apparatus for analyzing karyotype according to claim 2, wherein the apparatus further comprises: the top-down feature fusion is carried out in a down-sampling mode, and the feature graph of each level is sent back to the next level of feature graph after being down-sampled.

4. The method for using the apparatus for analyzing karyotype with automatic fitting of microscopic focusing plane according to claim 1, wherein: after feature extraction and feature fusion, the community detection network directly predicts the corresponding detection frame through convolution by adopting a yolov5 mode for the target frame prediction layer, and then automatically amplifies and inputs to obtain a local area to be amplified.

5. The method of using the apparatus for karyotyping according to claim 4, wherein the apparatus further comprises:

the automatic amplification input is to uniformly divide the image, match each detection frame with the area where the detection frame is located, acquire the minimum circumscribed rectangle and the direction of each detection frame, and expand the minimum circumscribed rectangle to the image boundary according to the corresponding direction, so as to obtain a local area to be amplified;

intercepting the part of the original input corresponding image, amplifying the part according to the proportion, performing feature extraction and recursive fusion through the community detection network again, and mapping the corresponding target frame back to the original input image after obtaining the target frame of the corresponding local area in the target frame prediction layer.

6. The method for using the apparatus for analyzing karyotype with automatic fitting of microscopic focusing plane according to claim 1, wherein: and the space fitting is to determine the z value when all the pre-scanning points are clearest, and then, interpolation is carried out according to the whole plane to obtain the corresponding z values of other points except all the pre-scanning points on the plane, so that the focal plane completely covering all the pre-scanning areas is obtained.

7. The method of using the apparatus for karyotyping according to claim 1, wherein the apparatus further comprises: the number of the pre-scanning points is 25.