CN115047610A

CN115047610A - Chromosome karyotype analysis device and method for automatically fitting microscopic focusing plane

Info

Publication number: CN115047610A
Application number: CN202210984995.4A
Authority: CN
Inventors: 韦然; 宋宁; 晏青; 吕明; 马伟旗; 唐悦
Original assignee: Hangzhou Daigens Biotech Ltd
Current assignee: Hangzhou Daigens Biotech Ltd
Priority date: 2022-08-17
Filing date: 2022-08-17
Publication date: 2022-09-13
Anticipated expiration: 2042-08-17
Also published as: CN115047610B

Abstract

The invention provides a chromosome karyotype analysis device and method for automatically fitting a microscopic focusing plane, wherein the device comprises a panoramic camera carrying a narrow-band filter with a specific waveband, a microscope provided with 10-fold and 100-fold objective lenses, a high signal-to-noise ratio industrial camera, a motor and an intelligent control center, a colony coordinate in a visual field is obtained on a clearest image through a colony detection network by obtaining an initial global image of a slide, a z value corresponding to a maximum value on each pre-scanning point in a definition fractional space is taken, space fitting is carried out to obtain a final z value focal plane, the optimal effect of the full-slide focal plane fitting is obtained on the premise of not increasing time consumption, accurate focusing of subsequent microscopic shooting can be guided, and efficient shooting is realized.

Description

Chromosome karyotype analysis device and method for automatically fitting microscopic focusing plane

Technical Field

The invention relates to the technical field of chromosome karyotype analysis, in particular to a chromosome karyotype analysis device and method capable of automatically fitting a microscopic focusing plane.

Background

Photomicrographs have been widely used in histopathology, cytology, and microscopic morphology observation and study of genetic chromosomes. With the continuous development of scientific technology, the degree of computer analysis has appeared, but most of chromosome examinations still rely on the traditional method, and the success or failure of photomicrography directly affects the correct analysis of chromosomes.

First, when fitting a focal plane before microscopic shooting of a chromosome colony, the conventional apparatus simply selects 25 pre-scan points of an overlay slide globally by using a slide, as shown in fig. 1, which may cause two problems: firstly, focusing failure or large focusing error is caused by the prospect of non-nuclear community and the like in the field of view of a pre-scanning point; ② the pre-scanning of the full slide consumes much time.

Secondly, when focusing is performed on scanning points one by one, the conventional equipment usually adopts a general focusing method, but a slide used for diagnosing the karyotype of the chromosome often has a large amount of impurity backgrounds, as shown in fig. 2, if direct focusing is performed, the clearest z value (longitudinal coordinate) corresponding to the karyotype community of the scanning points may not be obtained, and the error of each pre-scanning point will be accumulated to the subsequent focusing plane fitting, so that the shooting effect of the subsequent 10-time objective lens lower community whole-slide scanning is significantly influenced.

Thirdly, even if the most clear z value corresponding to the scanning point karyotype community is obtained and then the focusing plane fitting is performed, the existing equipment often adopts a simple interpolation method of the optimal z value point, which may cause the following problems: each pre-scanning point adopts a single optimal z value point, so that the overall influence of the error of the single z value point when the focal plane is fitted is enlarged.

Disclosure of Invention

In order to overcome the defects in the prior art, the invention provides a chromosome karyotype analysis device and method for automatically fitting a microscopic focusing plane.

In order to achieve the above object, the present invention provides a chromosome karyotype analysis apparatus for automatically fitting a microscopic focusing plane, comprising a microscope and a motor both electrically connected to an analysis control center, wherein the analysis control center is used for data receiving and processing, the microscope is provided with a stage and an objective lens, the objective lens is provided with a camera, a slide to be analyzed is arranged on the stage, the motor is connected to the stage, and there is no relative displacement in the vertical direction between the motor and the slide to be analyzed, and the motor controls the relative movement between the objective lens and the slide to be analyzed; (ii) a The motor adjusts the movement of the slide to be analyzed in the horizontal direction through the push-pull piece.

The invention provides a method for automatically fitting a chromosome karyotype analysis device for automatically fitting a microscopic focal plane, which comprises the following steps:

acquiring an initial global image of the slide, and acquiring an inscribed rectangle with the largest area for covering the distribution of the karyotype community according to the distribution condition of the chromosome karyotype community;

setting a plurality of pre-scanning points which are uniformly distributed on the obtained inscribed rectangle with the largest area, carrying out focusing change shooting on the pre-scanning points one by one under a 10-time mirror to obtain a plurality of images, and determining the clearest image and a z value corresponding to the longitudinal coordinate of the objective table by utilizing a definition evaluating algorithm;

obtaining community coordinates in a visual field on the clearest image through a community detection network, focusing again by taking the community coordinates as a focusing area, reducing the focusing variation degree, photographing again and evaluating the definition to obtain definition scores and corresponding z values of a plurality of chromosome images;

selecting the first n (n is more than or equal to 2) clearest chromosome images, and performing spatial coordinate interpolation on all pre-scanning points by adopting definition values and corresponding z values to obtain a definition score space with the thickness of n and a z value space with the thickness of n;

taking a z value corresponding to the maximum value on each pre-scanning point in the definition fractional space, and carrying out space fitting to obtain a final z value to a focal plane;

the community detection network performs feature extraction by taking a lightweight backbone network mobilenet _ v3 as a feature extraction layer to obtain output feature maps corresponding to three levels close to the output layer in the backbone network, wherein the feature extraction layer comprises two modes of feature fusion from bottom to top and feature fusion from top to bottom, so that recursive fusion of the feature maps is realized.

Preferably, the community detection network performs feature extraction by taking a lightweight backbone network mobilenet _ v3 as a feature extraction layer to obtain output feature maps corresponding to three levels close to the output layer in the backbone network, and the feature extraction layer includes two modes of feature fusion from bottom to top and feature fusion from top to bottom, so as to realize recursive fusion of the feature maps.

Preferably, the bottom-up feature fusion enlarges the small feature map of the lower layer to the same size as the feature map of the previous level in an upsampling mode, so as to realize the fusion of the stronger semantic features of the lower layer into the feature map of the upper layer.

Preferably, the top-down feature fusion is performed by means of downsampling, and the feature map of each stage is downsampled and then sent back to the next-layer feature map.

Preferably, after feature extraction and feature fusion, the community detection network directly predicts the corresponding detection frame through convolution by adopting a yolov5 mode for the target frame prediction layer, and then automatically amplifies and inputs the detection frame to obtain the local area to be amplified.

Preferably, the automatic amplification input is to uniformly divide the image, match each detection frame with the region where the detection frame is located, acquire the minimum circumscribed rectangle and the direction of each detection frame, and expand the minimum circumscribed rectangle to the image boundary according to the corresponding direction, so as to obtain the local region to be amplified; intercepting the part of the original input corresponding image, amplifying the part according to the proportion, performing feature extraction and recursive fusion through the community detection network again, and mapping the corresponding target frame back to the original input image after obtaining the target frame of the corresponding local area in the target frame prediction layer.

Preferably, the spatial fitting is to determine the clearest z value of all the pre-scanning points, and then perform interpolation according to the whole plane to obtain the corresponding z values of other points except all the pre-scanning points on the plane, so as to obtain the focal plane completely covering all the pre-scanning areas.

Preferably, the number of the pre-scanning points is 25.

The invention has the following beneficial effects:

1. the method has the advantages that the panoramic camera carrying the narrow-band filter with the specific wave band is additionally arranged to obtain the global view of the slide, so that the dense distribution condition of the chromosome colony is obtained in advance, the effective dense distribution range of the chromosome colony is selected for laying the pre-scanning points, the condition that the pre-scanning cannot be effectively focused due to less background during focusing can be effectively avoided, and meanwhile, the scanning time of the subsequent 10 times of the objective lens full slide can be reduced by reducing the scanning range;

2. a community coordinate in a visual field is obtained through a lightweight community detection network, a focusing area is determined according to the community coordinate, and definition evaluation is performed, so that a definition evaluation result of a background of a chromosome community position is obtained, and two benefits can be brought: focusing is carried out by a chromosome community, each pre-scanning point can obtain more accurate pre-scanning focusing precision, so that a focusing target is directly associated with a shooting target, and the influence of redundant background and the like is reduced; secondly, due to the fact that the influence of the background is removed, the measurement objects are unified, the condition that the reference z value and the definition score can be directly compared is obtained between every two pre-scanning points, and therefore the focusing plane established based on every pre-scanning point is more accurate;

3. by integrating the z value and the definition scoring, the optimal effect of the full-slide focal plane fitting is achieved on the premise of not increasing time consumption, accurate focusing of subsequent microscopic shooting can be guided, and efficient shooting is achieved.

Drawings

FIG. 1 is a schematic diagram of a prior art apparatus for setting up 25 pre-scan points globally across a slide;

FIG. 2 is a diagram showing the distribution of chromosome communities and impurities in a visual field when the existing device performs microscopic photography, wherein discrete origins are impurities, and clustered small spots are medium chromosome communities;

FIG. 3 is a schematic diagram showing the arrangement of points for pre-scanning in an effective dense distribution range of chromosome communities according to the present invention; FIG. 4 is a schematic diagram of determining a focusing area by obtaining a community coordinate in a field of view through a lightweight community detection network according to the present invention;

FIG. 5 is a schematic diagram of an inscribed rectangle for obtaining distribution of a covered karyotype community according to specific distribution of the slide karyotype community, wherein a diagram A is an image of a panoramic camera carrying a narrow-band filter with a specific wave band for photographing a slide, a diagram B is a state of the image after sharpening pretreatment, and diagrams C and D are front and back state diagrams of a maximum inscribed rectangle for obtaining distribution of the covered karyotype community;

FIG. 6 is a schematic diagram of a community detection network;

FIG. 7 is a schematic diagram of a plan for four "local regions" for magnification;

FIG. 8 is an exemplary diagram of focal plane fitting in the present invention;

FIG. 9 is a diagram illustrating the results of a fit to the focal plane of a pre-scan region;

FIG. 10 is a view showing an overall configuration of an apparatus for analyzing a karyotype based on an auto-fitting microscopic focal plane according to the present invention;

FIG. 11 is a logic diagram of a sharpness evaluation algorithm according to the present invention.

Detailed Description

To better illustrate the objects, aspects and advantages of the present invention, the present application will be further described with reference to specific examples.

Example 1:

the invention provides a chromosome karyotype analysis device capable of automatically fitting a microscopic focusing plane, as shown in FIG. 10, the device comprises a microscope 200 and a motor 300 which are electrically connected with an analysis control center 100, the analysis control center 100 is used for image receiving, definition analysis of a microscopic shooting diagnosis instrument and data receiving and processing of a vertical coordinate of the motor 300, such as a Metasight (MetaSigt) full-automatic cell microscopic image scanning system, the microscope 200 is provided with a carrier 201 and an objective 202, a slide to be analyzed is arranged on the carrier 201, a camera is arranged on the objective 202 to realize shooting of the slide to be analyzed, the motor 300 is connected with the carrier 201, no relative displacement exists between the motor 300 and the slide to be analyzed in the vertical direction, the motor 300 controls the vertical movement of the carrier 201 to realize adjustment of the relative displacement between the objective 202 and the slide to be analyzed, preferably, the motor 300 selects a stepping motor, the slide to be analyzed had a specification of 75 mm. times.25 mm. times.1.1 mm. The motor 300 is also connected with a push-pull member, and the horizontal movement control of the slide to be analyzed is realized through the push-pull member, which is arranged in CN 112859316A.

The use method of the device can be summarized as the following processes:

determining chromosome community distribution condition and laying pre-scanning points in a targeted manner

Carrying a narrow-band filter with a specific waveband through a panoramic camera to photograph the slide so as to obtain an initial panoramic picture of the slide; highlighting the specific distribution of the karyotype community through image preprocessing methods such as sharpening and the like, and obtaining an inscribed rectangle with the largest area for covering the distribution of the karyotype community according to the distribution condition;

25 pre-scanning points of 5x5 lines and columns which are uniformly distributed are arranged on the obtained inscribed rectangle with the largest area, continuous coarse grain (0.02 micron) z value variation shooting within the range of 0.4 micron of the lower stroke of a low-power mirror (10-power mirror) is carried out on the pre-scanning points one by one, and the clearest z value and the corresponding image are obtained through a definition evaluation algorithm;

obtaining the coordinates of the colony in the visual field on a clear image through a colony detection network (as shown in fig. 6), directly determining a focusing area by the coordinates of the colony, and acquiring the vicinity of a z value in a first stage on the basis of the focusing area (a)

0.05 micron) to further evaluate the fine-grained (0.005 micron) z value variable definition, thereby obtaining the definition scores of multiple chromosome images on the definition threshold and the corresponding z values.

Referring to fig. 11, the above sharpness evaluation algorithm can be summarized as the following process: the method comprises the steps of performing Gaussian blur processing on an image to be evaluated once to obtain a degraded image of the image, then comparing the change conditions of adjacent pixel values of an original image and the degraded image, determining the definition value according to the change conditions, wherein the smaller the calculation result is, the clearer the image is, and the clearer the image is, otherwise, the more fuzzy the image is.

Improved community detection network for improving chromosome community detection precision

The input image resolution of colony detection is relatively high (1600 x 1100), however, the detected chromosome colony is often relatively small (about 80x 80), and the original image is directly input into the detection network, so that the target is relatively small, and relatively high missing detection and false detection rates exist, which can significantly affect the above-mentioned focusing effect of determining the "focusing area" based on the colony detection result. On one hand, chromosome community detection needs to involve chromosome community detection under various medical scenes such as peripheral blood, amniotic fluid and bone marrow, wherein the number of chromosome communities contained in peripheral blood and amniotic fluid samples is large, and certain tolerance is provided for community omission detection, but the number of chromosome communities in bone marrow samples is small, so that the community detection sensitivity is required to be high. On the other hand, the background of the image where the chromosome community is located is complex, and a large number of impurities and the chromosome community often have high similarity under a large visual field, so that if the false detection rate of the community detection network is too high, the determination of the 'focusing area' based on the community detection result is greatly interfered by background noise. Therefore, in order to improve the accuracy of chromosome community detection, the invention mainly improves from two aspects:

increasing recursive feature fusion, and improving the detection capability of small targets:

the community detection network obtains output feature maps corresponding to three levels (stages) close to an output layer in the backbone network by taking a lightweight backbone network mobilenet _ v3 as a feature extraction layer. In the bottom-up feature fusion process, the small feature map of the lower layer is amplified to the same size as the feature map of the previous stage in an up-sampling mode, so that stronger semantic features (beneficial to classification) of the lower layer are fused into the feature map of the upper layer, the semantic expression on the feature map of each stage is enhanced, and the up-sampling method can be realized by using nearest neighbor difference values. Furthermore, the invention sends the feature map of each stage back to the next layer of feature map after down sampling, thereby realizing the recursive fusion of the feature maps, and the advantage of doing so is that the high-resolution information (beneficial to positioning) of the upper layer is fused into the feature map of the lower layer, thereby also enhancing the positioning information on the feature map of each stage, and simultaneously reducing the number of model parameters compared with the mode of additionally increasing the 'top-down feature fusion', thereby accelerating the detection speed of the community.

II, a two-stage community detection network based on a local automatic amplification mechanism:

after the feature extraction and the feature fusion, the mode of yolov5 is adopted at the target frame prediction layer to directly predict the corresponding detection frame through convolution, an automatic amplification input module is added on the basis of the method, firstly, whether the input is the first input is judged through counting judgment logic, and if the input is the first input, the detection result is regarded as an initial target frame.

After all the initial target frames are obtained, the image is divided into four regions according to the center of the image, then each initial target frame is distributed to the corresponding region, and meanwhile, the distribution rule of the initial target frames in the multiple regions is the region distributed with the area ratio of the region. After the "initial target frames" corresponding to the four regions are obtained, the minimum bounding rectangle and the direction thereof are obtained according to each target frame, and then 100 pixels are expanded (or expanded to the boundary) in four directions, namely, the upper direction, the lower direction, the left direction and the right direction, so as to obtain four to-be-amplified "local regions".

After four local areas are obtained, the original input corresponding image part is intercepted, then the proportion is kept (the proportion corresponding to 16x11 is obtained by adding a border), then the proportion is amplified to 1600x1100, the feature extraction and fusion module is input again, after a target frame corresponding to the local area is obtained through a target frame prediction layer, the corresponding target frame is mapped back to the original input image, and at the same time, the final detection result is directly output after the judgment of counting judgment logic is negative.

Fitting of spatial focal planes

Determining the clearest number n (taking 3 by default) in the obtained z values and the definition scores corresponding to different clear images of the number of the images under 25 pre-scanning points, then obtaining a definition score space with the thickness of n and a z value space with the thickness of n according to the spatial coordinate interpolation of the 25 points, and taking a z value corresponding to the maximum value on each coordinate point in the definition score space with the thickness of n for spatial fitting, thereby obtaining the final z value to the focal plane.

The actual image definition evaluating algorithm is used for measuring the 'unblurry degree' of an image, the image definition has a relatively accurate standard, but the image is difficult to accurately describe due to the fact that the image is not clear, the actual image definition evaluating algorithm is embodied as a clearer image, the absolute value and the relative value of the definition are always relatively large in reference meaning, and the absolute value and the relative value of the definition of the clearer image are relatively small in reference meaning.

The focal plane fitting mode can be summarized as that a z value of the clearest point is obtained by focusing and performing definition evaluation through 25 pre-scanning, and then the z value is taken as a representative of the point, so that interpolation is performed according to the whole plane to obtain the corresponding z values of other points except the 25 pre-scanning points on the plane.

As shown in fig. 8, specifically, there are four pre-scanning points a, B, C, and D, each coordinate point, for example, a (0.205,95), takes the z value and the corresponding sharpness score corresponding to the clearest three pictures, now a coordinate point E is given, falls into the ABCD area, and the abscissa and ordinate ratios from the four points ABCD are shown as follows, in order to find the z value corresponding to the E point:

a simpler method is to directly take the clearest z value of the ABCD three points to carry out interpolation so as to directly obtain the z value of the E point, and the calculation process is as follows:

(1) firstly, calculating a temporary point X parallel to the point E in the AD point connecting line, and calculating to obtain a z value as follows:

(2) then calculating a temporary point Y parallel to the point E in the BC point connecting line, and calculating to obtain a z value as follows:

(3) then calculating the z value of the E point according to the X point and the Y point as follows:

by the pixel-by-pixel traversal interpolation method, a focal plane completely covering 25 pre-scanning areas can be obtained, and the final effect is shown in fig. 9.

The method only takes the threshold value of one clearest image of each pre-scanning point, is easily interfered by a definition evaluation algorithm to random errors of a single picture, and does not fully utilize useful information in the process of focusing each pre-scanning point for multiple times, so that the error between a z-value point obtained by final interpolation and an actual z-value point is larger. Therefore, the invention carries out comprehensive interpolation on each pre-scanning point by taking the z value and the definition score of the clearest sheet number n (3 sheets are taken as a default), and the specific calculation process is as follows:

(1) firstly, calculating a temporary point X parallel to the point E in the AD point connecting line, and calculating to obtain a z value and a definition score corresponding to the clearest 3 graphs:

(2) then calculating a temporary point Y parallel to the point E in the BC point connecting line, and calculating to obtain a z value and a definition score corresponding to the clearest 3 graphs:

(3) then calculating the z value and the definition score of the clearest 3 graphs of the E point according to the X point and the Y point:

and taking the z value point with the highest definition score, namely 0.205625 corresponding to 93.500 as the final z value of the E point.

Finally, it should be noted that the above embodiments are only used for illustrating the technical solutions of the present invention and not for limiting the protection scope of the present invention, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions can be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A chromosome karyotype analysis device capable of automatically fitting a microscopic focusing plane is characterized in that: the slide analysis device comprises a microscope and a motor, wherein the microscope and the motor are electrically connected with an analysis control center, the analysis control center is used for receiving and processing data, the microscope is provided with an objective lens and a stage, a camera is arranged on the objective lens, a slide to be analyzed is arranged on the stage, the motor is connected with the stage, no relative displacement exists between the motor and the slide to be analyzed in the vertical direction, and the motor controls the relative movement between the objective lens and the slide to be analyzed; the motor adjusts the movement of the slide to be analyzed in the horizontal direction through the push-pull piece.

2. The method for automatically fitting a karyotype analysis apparatus for automatically fitting a microscopic focusing plane according to claim 1, wherein: comprises the following steps of (a) carrying out,

acquiring an initial global image of the slide, and acquiring an inscribed rectangle with the largest distribution area of the coverage karyotype community according to the distribution condition of the chromosome karyotype community;

3. The method for automatically fitting a karyotype analysis apparatus for automatically fitting microscopic focal planes according to claim 2, wherein: the bottom-up feature fusion enlarges the small feature map of the lower layer to the same size as the feature map of the previous level in an up-sampling mode, and realizes the fusion of the stronger semantic features of the lower layer into the feature map of the upper layer.

4. The method for automatically fitting a karyotype analysis apparatus for automatically fitting microscopic focal planes according to claim 3, wherein: the top-down feature fusion is carried out in a down-sampling mode, and the feature graph of each level is sent back to the next level of feature graph after being down-sampled.

5. The method for automatically fitting a karyotype analysis apparatus for automatically fitting microscopic focal planes according to claim 2, wherein: after feature extraction and feature fusion, the community detection network directly predicts the corresponding detection frame through convolution by adopting a yolov5 mode for the target frame prediction layer, and then automatically amplifies and inputs to obtain a local area to be amplified.

6. The method for automatically fitting a karyotype analysis apparatus for automatically fitting microscopic focal planes according to claim 5, wherein:

the automatic amplification input is to uniformly divide the image, match each detection frame with the area where the detection frame is located, acquire the minimum circumscribed rectangle and the direction of each detection frame, and expand the minimum circumscribed rectangle to the image boundary according to the corresponding direction, so as to obtain a local area to be amplified;

intercepting the part of the original input corresponding image, amplifying the part according to the proportion, performing feature extraction and recursive fusion through the community detection network again, and mapping the corresponding target frame back to the original input image after obtaining the target frame of the corresponding local area in the target frame prediction layer.

7. The method for automatically fitting a karyotype analysis apparatus for automatically fitting microscopic focal planes according to claim 2, wherein: and the space fitting is to determine the z value when all the pre-scanning points are clearest, and then carry out interpolation according to the whole plane to obtain the corresponding z values of other points except all the pre-scanning points on the plane, thereby obtaining the focal plane completely covering all the pre-scanning areas.

8. The method for automatically fitting a karyotype analysis apparatus for automatically fitting microscopic focal planes according to claim 2, wherein: the number of the pre-scanning points is 25.