WO2022095082A1

WO2022095082A1 - Micromanipulation platform three-dimensional positioning method for cell injection

Info

Publication number: WO2022095082A1
Application number: PCT/CN2020/127759
Authority: WO
Inventors: 汝长海; 郝淼; 陈瑞华; 翟荣安; 岳春峰; 孙钰
Original assignee: 江苏集萃微纳自动化系统与装备技术研究所有限公司
Priority date: 2020-11-04
Filing date: 2020-11-10
Publication date: 2022-05-12
Also published as: CN112101575B; CN112101575A

Abstract

A micromanipulation platform three-dimensional positioning method for cell injection, comprising: acquiring a sample picture and establishing a positive sample library and a negative sample library, the positive sample library comprising an injection needle positive sample library and a holding needle positive sample library; the negative sample library comprising an injection needle negative sample library and a holding needle negative sample library; respectively establishing a positive sample training set and a negative sample training set by means of the positive sample library and the negative sample library, and generating a support vector machine model by training; respectively moving an injection needle and a holding needle under a microscope along a preset route, and respectively capturing pictures; and using the trained support vector machine model to process the captured pictures, and determining whether the pictures are positive samples until the captured injection needle and holding needle pictures are all positive samples for completing aligning the injection needle with the holding needle. The micromanipulation platform three-dimensional positioning method for cell injection makes it possible to position during cell injection, and has the advantages of high positioning efficiency and high positioning accuracy.

Description

A three-dimensional positioning method of a micromanipulation platform for cell injection

technical field

The invention relates to the technical field of micro-operation computing, in particular to a three-dimensional positioning method of a micro-operation platform for cell injection.

Background technique

Since the 21st century, with the extensive application of artificial assisted reproductive technology and biomedicine, micromanipulation technology in the field of cell engineering has gradually become a cutting-edge technology that is inseparable from the development of human society. Micromanipulation technology refers to an operation method that uses an actuator with micro-nano motion precision to complete specific experimental tasks for controlled objects such as organisms, materials, and chemical molecules through feedback methods such as vision and force. Traditional micromanipulation techniques are all done manually and are generally operated by experienced laboratory personnel who have received at least two or three years of professional training, such as intracytoplasmic sperm injection, nuclear transfer, chimera technique, embryo transfer and Microdissection, etc., of which cell microinjection technology is the most widely used, the main applications include intracellular drug injection, nuclear transfer, embryo transfer and intracytoplasmic sperm injection (ICSI). During the microinjection process, the experimenter sits in front of the microscope and observes the working space with his eyes, and operates the micromanipulator to complete the experiment according to his own experience. Certain challenges. It can be seen that this process is time-consuming and labor-intensive, and at the same time, it is very dependent on the personal experience of the operator. It is difficult to ensure the consistency of all operating processes, which restricts the development of bioengineering technology and life sciences to a certain extent.

With the continuous advancement of machining technology and automation technology, it is the current research trend to replace manual labor with machines to improve the level of automation. In recent years, micromanipulation technology has made significant progress, and researchers have developed micromanipulation systems with various mechanical structures, driving methods and control methods, but these micromanipulation systems have not achieved wide applicability, especially However, there are still some problems in the method research of the three-dimensional positioning of cells, as follows.

On the one hand, when acquiring the two-dimensional position, there is a problem that the calibration accuracy of the microscopic vision system is not high. Since the objects of micro-operation are all in the micron level, the movement precision of the operation platform is very high. At present, the precision of micromanipulation mainly depends on the precision of machinery. However, due to the inevitable errors and installation errors of mechanical equipment, the accuracy of micromanipulation will be reduced, and even the results of micromanipulation will be directly affected. . At present, mechanical auxiliary instruments are often used to assist installation to reduce the installation error accuracy. This error compensation method requires repeated measurements and is accompanied by a large amount of data processing. Additional errors will be added. Therefore, since there is no effective error compensation method, the current micromanipulation without exception still relies on pure manual or semi-automatic operation, which obviously cannot meet the high-efficiency and high-quality operation requirements of modern intelligent medical technology.

On the other hand, when acquiring the depth position, there is still a problem that the depth information cannot be accurately acquired. The acquisition of depth information is a challenging problem in the field of micromanipulation robotics. There are basically two ways to acquire depth information at present: the first is indirect acquisition method. Based on the image processing technology, the depth information of the target cells is obtained using the contact detection algorithm. The usual practice of the contact detection algorithm is to move the injection needle directly above the target cell, lower the microinjection needle, and track the image changes in the area near the needle tip. Through visual feedback, when the needle touches the target cell, the cell surface will deform , then this position is calibrated as the zero point position of the depth coordinate. However, in this way, since the needle tip needs to directly contact the cell, it will deform, and the target tracking algorithm used in visual feedback (such as template matching method, SSD optical flow method, motion history image MHI and active contour model, etc.) Certain specificity cannot meet the real-time requirements of the operation. In addition, the needle tip is very easy to puncture the cell during the contact process, resulting in the failure of the microscopic operation. The second is the direct access method. Inspired by the macroscopic binocular stereo vision, it was introduced into the field of micromanipulation. By placing the microscopes in the horizontal and vertical directions, or using a stereo microscope to achieve binocular stereo vision, the depth information of the target cells can be obtained in this way, but this method is required for micron-level errors. Errors are relatively large, and ensuring accuracy remains a particular challenge.

SUMMARY OF THE INVENTION

In view of the deficiencies of the prior art, the purpose of the present invention is to provide a three-dimensional positioning method of a micromanipulation platform for cell injection with high efficiency and high precision. It adopts the following technical solutions:

A three-dimensional positioning method of a micromanipulation platform for cell injection, comprising:

Collect sample pictures and establish a positive sample library and a negative sample library; wherein, the positive sample library includes an injection needle positive sample library and a suction needle positive sample library, and the negative sample library includes an injection needle negative sample library and a suction needle negative sample library a sample library, wherein the needle tip in the positive sample library is in the focal plane of the microscope, and the needle tip in the negative sample library is not in the focal plane of the microscope;

Establish a positive sample training set and a negative sample training set respectively through the positive sample library and the negative sample library, and train to generate a support vector machine model;

Move the injection needle and the holding needle along the preset route under the microscope respectively, and take pictures respectively;

Use the trained support vector machine model to process the captured images, and determine whether the images are positive samples. When the captured images of injection needles and suction needles are both positive samples, the needle alignment of the injection needles and the suction needles is completed. .

As a further improvement of the present invention, a positive sample training set and a negative sample training set are respectively established through the positive sample library and the negative sample library, and a support vector machine model is generated by training, which specifically includes:

By calculating the gradients G _x (x, y), G _y (x, y) of each sample image in the sample library in the x-axis direction and the y-axis direction;

G _x (x,y)=H(x+1,y)-H(x-1,y)

G _y (x, y)=H(x, y+1)-H(x, y-1)

Where: H(x, y) represents the pixel value of the input sample image at the pixel point (x, y), G _x (x, y), G _y (x, y) represent the horizontal gradient and vertical gradient respectively ;

Divide the sample image into several uniform "Block Cells", and set the size and other parameters of the Cell to obtain the feature vector of the Cell, and then combine the Cell units into a large Block. A Block includes multiple Cells, and each Block contains The HOG feature can be obtained by concatenating the feature vectors of all Cells;

Combined with the method of machine learning, through the training of positive sample training set and negative sample training set, the training data set of support vector machine is obtained as:

T={(x ₁ , y ₁ ), (x ₂ , y ₂ ), ... (x _i , y _i )}

Among them, i=1, 2, 3,...N, N is the number of samples in the positive library or the number of samples in the negative library, x _i is the i-th feature vector, x _i ∈R ⁿ , y _i is the i-th eigenvector class labels, y _i ∈ {-1, +1}, when y _i is equal to +1, it is a positive sample library, and when it is -1, it is a negative sample library;

A support vector machine model is generated according to the support vector machine training data set.

As a further improvement of the present invention, the judging whether the picture is a positive sample specifically includes: judging whether the shot pictures of the injection needle and the holding needle are positive samples by calculating the HOG feature value.

As a further improvement of the present invention, before the collection of sample pictures and the establishment of a positive sample library and a negative sample library, the method further includes:

Perform the error correction of the micromanipulation platform.

As a further improvement of the present invention, the error correction of the micromanipulation platform specifically includes:

Place the ruler on the micro-operating platform, move the micro-operating platform in a fixed direction at a fixed step, and acquire the ruler image, and ensure that the two images before and after partially overlap;

Stitch the two images before and after the direction;

Calculate the systematic error of the micromanipulation platform in this direction;

Perform autonomous compensation for the system error in the direction to correct the system error in the direction;

Move the micromanipulation platform in other directions with fixed steps and acquire images respectively, calculate the systematic errors of the micromanipulation platform in other directions, and automatically compensate the system errors in other directions to complete the system in all directions. Error correction.

As a further improvement of the present invention, the calculation of the systematic error of the micromanipulation platform in this direction specifically includes:

Calculate the pixel spacing;

Calculate the actual displacement distance of the two images before and after in this direction according to the pixel spacing;

The systematic error in this direction is obtained according to the actual displacement distance.

As a further improvement of the present invention, the calculation of the pixel spacing specifically includes:

The pixel spacing is calculated using the formula S=M/N; wherein, S is the pixel spacing, M is the scale length, and N is the number of pixels within the M length.

As a further improvement of the present invention, the described actual displacement distance of two images before and after the direction is calculated according to the pixel spacing, specifically including:

The actual displacement distance is calculated using the formula AA _{1 actual} =S*AA ₁ ; wherein, AA ₁ is the actual displacement distance, and AA ₁ is the number of pixels relative to the displacement of the front and rear images in this direction.

As a further improvement of the present invention, the described system error of this direction is obtained according to the actual displacement distance, specifically including:

According to the formulas AA _{1 actual *} cos(θ) and AA _{1 actual *} sin (θ), the two components of the systematic error in this direction are obtained; where, θ is the declination angle between the image coordinate system and the micro-operating platform coordinate system.

As a further improvement of the present invention, the system error in the direction is autonomously compensated to correct the system error in the direction, specifically including:

Compensation calculation is performed through computer closed-loop feedback, and the system error in this direction is compensated autonomously to correct the system error in this direction.

Beneficial effects of the present invention:

The three-dimensional positioning method of the micro-operation platform for cell injection of the present invention can realize positioning during cell injection, and has the advantages of high positioning efficiency and high positioning accuracy.

The above description is only an overview of the technical solutions of the present invention, in order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand , the following specific preferred embodiments, and in conjunction with the accompanying drawings, are described in detail as follows.

Description of drawings

1 is a flow chart of a three-dimensional positioning method of a micromanipulation platform for cell injection in a preferred embodiment of the present invention;

Fig. 2 is the needle alignment schematic diagram of the completed injection needle and the suction needle in the preferred embodiment of the present invention;

Fig. 3 is the flow chart of performing the error correction of the micro-operation platform in the preferred embodiment of the present invention;

4 is a schematic diagram of a scale in a preferred embodiment of the present invention;

5 is a schematic diagram of two images before and after in the preferred embodiment of the present invention;

Fig. 6 is the splicing schematic diagram of two images before and after in the preferred embodiment of the present invention;

FIG. 7 is a schematic diagram of splicing two images before and after the positive X-axis in a preferred embodiment of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

As shown in FIG. 1 , the three-dimensional positioning method of the micromanipulation platform for cell injection in the preferred embodiment of the present invention includes the following steps:

Step S10, collect sample pictures and establish a positive sample library and a negative sample library; wherein, the positive sample library includes a positive sample library for injection needles and a positive sample library for suction needles, and the negative sample library includes a negative sample library for injection needles and suction needles. Hold the needle negative sample bank, the needle tip in the positive sample bank is in the focal plane of the microscope, and the needle tip in the negative sample bank is not in the focal plane of the microscope.

Step S20: Establish a positive sample training set and a negative sample training set respectively through the positive sample library and the negative sample library, and train to generate a support vector machine model.

Specifically, the extraction of histogram of orientation gradient (HOG) features is performed on the positive sample library and the negative sample library respectively, including:

First, by calculating the gradients G _x (x, y) and G _y (x, y) of each sample image in the sample library in the x-axis direction and the y-axis direction;

G _x (x,y)=H(x+1,y)-H(x-1,y)

G _y (x, y)=H(x, y+1)-H(x, y+1)

Then, divide the sample image into several uniform "Block Cells", and set the size and other parameters of the Cell to obtain the feature vector of the Cell, and then combine the Cell units into a large Block, a Block includes multiple Cells, each The HOG feature can be obtained by concatenating the feature vectors of all Cells in the block; among them, other parameters include bin, gradient direction, etc.

Then combined with the machine learning method, through the training of the positive sample training set and the negative sample training set, the training data set of the support vector machine is obtained as:

T={(x ₁ , y ₁ ), (x ₂ , y ₂ ), ... (x _i , y _i )}

Step S30, respectively move the injection needle and the suction needle along a preset route under the microscope, and take pictures respectively. Specifically, under the microscope, the injection needle and the holding needle are moved from above the focal plane to the negative direction of the Z-axis with a fixed step, or from below the focal plane to the positive direction of the Z-axis with a fixed step, and each step is taken for one shot. a picture.

Step S40, using the trained support vector machine model to process the photographed picture, and determine whether the picture is a positive sample, until the photographed pictures of the injection needle and the holding needle are both positive samples, that is, the injection needle and the holding needle are completed. 's needle.

Among them, by calculating the HOG feature value, it is determined whether the pictures of the injection needle and the holding needle are positive samples.

In one of the embodiments, the present invention further includes step A: performing error correction on the micromanipulation platform. In order to achieve high-precision two-dimensional positioning on the XY plane.

Specifically, step A specifically includes:

Step A1: Place the ruler on the micro-operating platform, move the micro-operating platform in a fixed direction at a fixed step, and acquire images of the ruler, and ensure that the front and rear images partially overlap.

In this embodiment, the scale is a two-dimensional plane scale, as shown in FIG. 4 . The two acquired images before and after are shown in Figure 5, namely the front frame and the back frame.

Step A2, stitching the front and rear images in the direction. The stitched image is shown in Figure 6.

Step A3: Calculate the systematic error of the micromanipulation platform in this direction. Specifically include:

Step A31: Calculate the pixel spacing. Specifically include:

Step A32, calculate the actual displacement distance of the front and back two images in this direction according to the pixel spacing; specifically include:

Step A33: Obtain the systematic error of the direction according to the actual displacement distance. Specifically include:

As shown in Figure 7, when the operating direction of the micromanipulation platform is the positive direction of the X-axis, the systematic error in this direction is +XΔ, which can be divided into two components: + _XΔX and + _XΔy , which satisfy the following formula:

+XΔ _X =AA _{1 actual} *cos(θ);

+XΔ _y =AA _{1 actual} *sin(θ);

+XΔ _X and +XΔ _y are the compensation values that need to be compensated when the X axis moves forward. In the same way, the compensation value can be obtained when moving in other directions.

Step A4: Perform autonomous compensation for the systematic error in the direction to correct the systematic error in the direction. Specifically include:

In this embodiment, performing the error correction of the micro-operation platform further includes:

Step A5: Move the micro-operating platform in other directions with a fixed step and acquire images respectively, calculate the systematic errors of the micro-operating platform in other directions, and perform autonomous compensation for the system errors in other directions to complete all the steps. Systematic error correction for orientation. Among them, all directions include the positive direction of the X axis, the negative direction of the X axis, the positive direction of the Y axis, and the negative direction of the Y axis.

In this embodiment, performing the error correction of the micro-operating platform further includes: taking multiple sets of images, and performing multiple calculations to obtain the average value of the systematic errors of the micro-operating platform in this direction. The system error calculation accuracy can be improved, and finally the error correction accuracy can be improved.

The method for obtaining the two-dimensional position in the three-dimensional positioning method of the micromanipulation platform for cell injection proposed by the present invention abandons the traditional manual error factors (such as translational motion parts, rotational motion parts, Rolling moving parts) separately perform data acquisition and calibration, and manually compensate for error correction. Based on image stitching technology, the current mechanical errors, CCD installation errors and pixel/micron conversion errors that affect the accuracy of microinjection are integrated and unified, without manual labor. Auxiliary, it can realize self-compensation and correction, and the error can be controlled at the pixel level.

In the three-dimensional positioning method of the micromanipulation platform for cell injection proposed by the present invention, for the two-dimensional position acquisition method, the mentioned system error autonomous compensation algorithm is not only applicable to the micromanipulation system, but also to other mobile platforms. The error correction is not only simple to operate, but also has the characteristics of high efficiency and high accuracy.

The method for obtaining the depth position in the three-dimensional positioning method of the micromanipulation platform for cell injection proposed by the present invention proposes a method for determining the depth position in the form of "focus plane to needle" based on HOG features combined with machine learning. The new method not only has high processing speed, but also has high accuracy, thus solving the problem of inaccurate depth position acquisition in the current positioning method.

The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or transformations made by those skilled in the art on the basis of the present invention are all within the protection scope of the present invention. The protection scope of the present invention is subject to the claims.

Claims

A three-dimensional positioning method for a micromanipulation platform for cell injection, characterized by comprising:

Collect sample pictures and establish a positive sample library and a negative sample library; wherein, the positive sample library includes an injection needle positive sample library and a suction needle positive sample library, and the negative sample library includes an injection needle negative sample library and a suction needle negative sample library a sample library, wherein the needle tip in the positive sample library is in the focal plane of the microscope, and the needle tip in the negative sample library is not in the focal plane of the microscope;

Establish a positive sample training set and a negative sample training set respectively through the positive sample library and the negative sample library, and train to generate a support vector machine model;

Move the injection needle and the holding needle along the preset route under the microscope respectively, and take pictures respectively;

Use the trained support vector machine model to process the captured images, and determine whether the images are positive samples. When the captured images of injection needles and suction needles are both positive samples, the needle alignment of the injection needles and the suction needles is completed. .
The three-dimensional positioning method of a micromanipulation platform for cell injection according to claim 1, wherein a positive sample training set and a negative sample training set are established respectively through the positive sample library and the negative sample library, and the training Generate a support vector machine model, including:

First, calculate the gradients G x (x, y) and G y (x, y) of each sample image in the sample library in the x-axis and y-axis directions;

G x (x,y)=H(x+1,y)-H(x-1,y)

G y (x, y)=H(x, y+1)-H(x, y-1)

Where: H(x, y) represents the pixel value of the input sample image at the pixel point (x, y), G x (x, y), G y (x, y) represent the horizontal gradient and vertical gradient respectively ;

Then, the sample image is divided into several uniform "BlockCell", and the size and other parameters of the Cell are set to obtain the feature vector of the Cell, and then the Cell units are combined into a large Block. A Block includes multiple Cells, and each Block The HOG feature can be obtained by concatenating the feature vectors of all Cells in the cell;

Then combined with the method of machine learning, through the training of positive sample training set and negative sample training set, the training data set of support vector machine is obtained as:

T={(x 1 , y 1 ), (x 2 , y 2 ), ... (x i , y i )}

Among them, i=1, 2, 3,...N, N is the number of samples in the positive library or the number of samples in the negative library, x i is the i-th feature vector, x i ∈R n , y i is the i-th eigenvector class labels, y i ∈ {-1, +1}, when y i is equal to +1, it is a positive sample library, and when it is -1, it is a negative sample library;

A support vector machine model is generated according to the support vector machine training data set.
The three-dimensional positioning method of a micromanipulation platform for cell injection according to claim 2, wherein the judging whether the picture is a positive sample specifically comprises: judging the shot of the injection needle and the suction needle by calculating the HOG characteristic value. Whether the needle-holding pictures are positive samples.
A three-dimensional positioning method for a micromanipulation platform for cell injection according to claim 1, characterized in that, before the collection of sample pictures and the establishment of a positive sample library and a negative sample library, the method further comprises:

Perform the error correction of the micromanipulation platform.
The three-dimensional positioning method of a micro-operating platform for cell injection according to claim 4, wherein the performing error correction of the micro-operating platform specifically includes:

Place the ruler on the micro-operating platform, move the micro-operating platform in a fixed direction at a fixed step, and acquire the ruler image, and ensure that the two images before and after partially overlap;

Stitch the two images before and after the direction;

Calculate the systematic error of the micromanipulation platform in this direction;

Perform autonomous compensation for the system error in the direction to correct the system error in the direction;

Move the micromanipulation platform in other directions with fixed steps and acquire images respectively, calculate the systematic errors of the micromanipulation platform in other directions, and automatically compensate the system errors in other directions to complete the system in all directions. Error correction.
The three-dimensional positioning method of a micromanipulation platform for cell injection according to claim 5, wherein the calculating the systematic error of the micromanipulation platform in this direction specifically includes:

Calculate the pixel spacing;

Calculate the actual displacement distance of the two images before and after in this direction according to the pixel spacing;

The systematic error in this direction is obtained according to the actual displacement distance.
A three-dimensional positioning method for a micromanipulation platform for cell injection according to claim 6, wherein the calculating the pixel spacing specifically includes:

The pixel spacing is calculated using the formula S=M/N; wherein, S is the pixel spacing, M is the scale length, and N is the number of pixels within the M length.
A kind of micromanipulation platform three-dimensional positioning method for cell injection as claimed in claim 7, it is characterised in that the actual displacement distance of the two images before and after the direction is calculated according to the pixel spacing, specifically includes:

The actual displacement distance is calculated using the formula AA 1 actual =S*AA 1 ; wherein, AA 1 is the actual displacement distance, and AA 1 is the number of pixels relative to the displacement of the front and rear images in this direction.
A kind of micromanipulation platform three-dimensional positioning method for cell injection as claimed in claim 8, is characterized in that, described according to actual displacement distance obtains the systematic error of this direction, specifically comprises:

According to the formulas AA 1 actual * cos(θ) and AA 1 actual * sin (θ), the two components of the systematic error in this direction are obtained; where, θ is the declination angle between the image coordinate system and the micro-operating platform coordinate system.
The three-dimensional positioning method of a micromanipulation platform for cell injection according to claim 5, wherein the autonomous compensation of the systematic error in the direction and the correction of the systematic error in the direction specifically include:

Compensation calculation is performed through computer closed-loop feedback, and the system error in this direction is compensated autonomously to correct the system error in this direction.