CN113048901A - Method for measuring nanoscale three-dimensional deformation under microscope based on optical flow algorithm - Google Patents
Method for measuring nanoscale three-dimensional deformation under microscope based on optical flow algorithm Download PDFInfo
- Publication number
- CN113048901A CN113048901A CN202110246918.4A CN202110246918A CN113048901A CN 113048901 A CN113048901 A CN 113048901A CN 202110246918 A CN202110246918 A CN 202110246918A CN 113048901 A CN113048901 A CN 113048901A
- Authority
- CN
- China
- Prior art keywords
- deformation
- plane displacement
- optical flow
- flow algorithm
- displacement field
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/16—Measuring arrangements characterised by the use of optical techniques for measuring the deformation in a solid, e.g. optical strain gauge
- G01B11/167—Measuring arrangements characterised by the use of optical techniques for measuring the deformation in a solid, e.g. optical strain gauge by projecting a pattern on the object
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10056—Microscopic image
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T90/00—Enabling technologies or technologies with a potential or indirect contribution to GHG emissions mitigation
Abstract
The invention relates to a method for measuring nanoscale three-dimensional deformation under a microscope based on an optical flow algorithm, which comprises the following steps: respectively acquiring an orthographic projection image before material deformation and an orthographic projection image after deformation; obtaining an in-plane displacement field between the two images before and after deformation by using an optical flow algorithm; and obtaining an out-of-plane displacement field of the material deformation from the in-plane displacement field according to the mathematical relation of the wedge surfaces. When an object is deformed under a microscope, the deformation of the solid material can be regarded as the wedge surface deformation, and the mathematical relation between the in-plane displacement and the out-of-plane displacement is obtained. The method can finish measurement by only shooting two images by one camera without interfering light paths, has strong applicability to test environment, is suitable for industrial detection, can greatly reduce measurement errors and measurement speed, has higher measurement precision, and is suitable for dynamic measurement.
Description
Technical Field
The invention relates to the technical field of industrial detection, in particular to a method for measuring nanoscale three-dimensional deformation under a microscope based on an optical flow algorithm, and specifically relates to a method for measuring nanoscale three-dimensional deformation under a microscope based on an optical flow algorithm.
Background
The detection technology of the nanoscale three-dimensional deformation of the solid material is always a difficult problem which is tried to overcome by scientific researchers. The three-dimensional deformation mainly refers to deformation displacement of an object in three directions of x (horizontal direction), y (vertical direction) and z (out-of-plane direction), wherein the displacement in the x and y directions is called in-plane displacement, and the displacement in the z-axis direction is called out-of-plane displacement. Currently, methods for measuring in-plane displacement include digital image correlation, block motion matching estimation, and the like, and methods for measuring out-of-plane displacement include a projected grating method, an interference fringe method, a microscopic scanning method, and the like. However, the measurement algorithm for the in-plane displacement is not sensitive to the out-of-plane displacement, and cannot realize the measurement of three-dimensional deformation in one step, but the device in the measurement method for the out-of-plane displacement is generally complex, the measurement time of the scanning method is too long, the requirements of the fringe and projection methods on the environment are high, phase unwrapping operation is required, the error is large, the precision is low, and the method is not suitable for industrial detection.
Optical flow algorithm (Optical flow) is an image recognition method in signalization, and is very useful in the fields of pattern recognition, computer vision and other image processing, and can be used for motion detection, object cutting, calculation of collision time and object expansion, motion compensation coding, or stereo measurement through object surface and edge, and the like. However, the prior art does not adopt the precedent of adopting an optical flow algorithm to measure the nanoscale deformation.
Disclosure of Invention
The invention mainly aims to provide a method for measuring nanoscale three-dimensional deformation under a microscope based on an optical flow algorithm, and the technical problem to be solved is that the nanoscale three-dimensional deformation of a solid material is difficult to measure in the prior art.
The purpose of the invention and the technical problem to be solved are realized by adopting the following technical scheme. The invention provides a method for measuring nanoscale three-dimensional deformation under a microscope based on an optical flow algorithm, which comprises the following steps:
respectively acquiring an orthographic projection image before material deformation and an orthographic projection image after deformation;
obtaining an in-plane displacement field between the two images before and after deformation by using an optical flow algorithm;
and obtaining an out-of-plane displacement field of the material deformation from the in-plane displacement field according to the mathematical relation of the wedge surfaces.
The object of the present invention and the technical problems solved thereby can be further achieved by the following technical measures.
Preferably, in the method for measuring the nanoscale three-dimensional deformation under the microscope based on the optical flow algorithm, the optical flow algorithm is used for obtaining the in-plane displacement field between the two images before and after deformation; according to the mathematical relationship of the wedge surfaces, the out-of-plane displacement field of the material deformation is obtained by the in-plane displacement field, and the method specifically comprises the following steps:
regarding the deformation as wedge surface deformation, according to the mathematical relationship of the wedge surfaces, the out-of-plane displacement w of the material deformation is calculated by the formula (1):
w＝R sinθ (1)
in formula (1), R is the image width; θ is a deformed wedge angle, which is calculated by equation (2):
in the formula (2), d is an in-plane displacement field between two images before and after changing; r is the image width and is known; theta is a deformation wedge angle; u and v are two components of an in-plane displacement field between two images before and after deformation respectively, wherein the u and v are obtained by calculation through an optical flow algorithm.
Preferably, the optical flow algorithm is a Horn-Schunck optical flow algorithm, and u and v are obtained by calculating the Horn-Schunck optical flow algorithm and are obtained by iterating the formula (3) and the formula (4) respectivelyn+1And vn+1：
In the formulae (3) and (4),
and
the average values of u and v in the sub-region centered on (x, y), respectively, and α is a smoothing parameter.
Preferably, in the method for measuring nanoscale three-dimensional deformation under a microscope based on the optical flow algorithm, the in-plane displacement of all points between two images before and after deformation is less than 3 pixels.
Preferably, the aforementioned method for measuring nanoscale three-dimensional deformation under a microscope based on an optical flow algorithm, wherein the size of each of the two images is between 256 × 256 and 512 × 512.
Preferably, the method for measuring the nanoscale three-dimensional deformation under the microscope based on the optical flow algorithm, wherein the obtaining of the in-plane displacement field between the two images before and after deformation by using the optical flow algorithm, includes:
firstly, an in-plane motion field between two images before and after deformation is carried out by utilizing an optical flow algorithm, then the obtained in-plane motion field is multiplied by a time factor to obtain an in-plane displacement field of three-dimensional deformation, and then the unit of out-of-plane displacement is converted into a length unit from a pixel unit according to the magnification of a microscope to obtain the out-of-plane displacement size, wherein the time factor is the time difference between the two images before and after deformation.
Preferably, the method for measuring the nanoscale three-dimensional deformation under the microscope based on the optical flow algorithm, wherein the obtaining of the image before the material deformation and the orthographic projection image after the material deformation respectively comprises:
the method comprises the following steps of coaxially placing and packaging a light source, a three-dimensional light-transmitting object stage, a microscope and an industrial camera in a dark box, starting the light source, placing a solid material to be measured on the light-transmitting object stage above the light source, and adjusting the magnification factor of the microscope and the focal length of the industrial camera to enable an image of the upper surface of the solid material to be clearly shot by the industrial camera;
the orthographic projection image of the solid material before deformation is shot, then the shooting time interval is set according to the deformation speed, and the orthographic projection image of the solid material after deformation is shot.
Preferably, the method for measuring nanoscale three-dimensional deformation under a microscope based on an optical flow algorithm is described, wherein the microscope is a pure optical magnifying microscope.
Preferably, the method for measuring the nanoscale three-dimensional deformation under the microscope based on the optical flow algorithm further includes:
and establishing a rectangular coordinate system by taking the position of the deformation generating source as an original point, and judging the vector direction of the out-of-plane displacement field according to the quadrant where the measuring point is located, the symbol of the horizontal component and the symbol of the vertical component of the in-plane displacement field.
Preferably, the method for measuring the nanoscale three-dimensional deformation under the microscope based on the optical flow algorithm is implemented by processing Matlab software.
By means of the technical scheme, the method for measuring the nanoscale three-dimensional deformation under the microscope based on the optical flow algorithm at least has the following advantages:
1. firstly, obtaining an in-plane displacement field between two images before and after deformation by using an optical flow algorithm; and then obtaining an out-of-plane displacement field of the material deformation from the in-plane displacement field according to the mathematical relation of the wedge surfaces. According to researches, when an object is deformed under a microscope, the deformation of the solid material can be regarded as wedge surface deformation, and under the model of the wedge surface deformation, the in-plane displacement and the out-of-plane displacement have a certain mathematical relationship, so that the scalar field of the out-of-plane displacement field can be obtained by measuring the in-plane displacement field of the object through an optical flow method according to the mathematical relationship of the in-plane displacement and the out-of-plane displacement.
2. According to the method, all material deformation in a certain range is regarded as wedge surface deformation, and the out-of-plane displacement scalar field of the deformation can be obtained from the in-plane displacement field according to the mathematical relationship of the wedge surfaces.
3. The optical flow algorithm can provide an in-plane motion field when a certain object in two images moves within a certain time interval, and the in-plane motion field is multiplied by the time interval, namely a displacement field, so that the optical flow algorithm can realize the in-plane displacement measurement of the object after calibration. Compared with other in-plane displacement measurement methods, the optical flow algorithm can measure only when the images before and after deformation meet the strength consistency assumption and the gradient assumption, has a narrow deformation range and high precision, and is particularly suitable for measuring nano-scale deformation. Meanwhile, the method introduces a time factor and is suitable for dynamic measurement.
4. The method can finish measurement by only shooting two images by one camera without interfering a light path, has strong applicability to a test environment, and is suitable for industrial detection. Meanwhile, the method does not need to convert to a frequency domain or perform phase unwrapping operation, can greatly reduce measurement errors, has higher measurement precision, and is suitable for dynamic measurement.
5. The method not only can be used for measuring the out-of-plane displacement of the micro-nano deformation of the material, but also can be used for measuring the out-of-plane displacement of the large-size deformation of the material, the size of the image is set to be 256 × 256 to 512 × 512 in order to guarantee the calculation time and the measurement precision, the in-plane displacement fields of the two images are pretested, and the in-plane displacement of all points between the two images before and after deformation is guaranteed to be less than 3 pixels.
6. The method further establishes a rectangular coordinate system by taking the position of the deformation generating source point as an origin, and judges the vector direction of the out-of-plane displacement field according to the symbol of the horizontal component and the symbol of the vertical component of the displacement field in the quadrant where the measuring point is positioned and the in-plane displacement field.
7. The method can be used for measuring the out-of-plane displacement of the qualitative deformation of the material and the uniformity of the deformation of the material, has the characteristics of high measuring speed, high measuring precision, good repeatability and the like, can realize dynamic measurement, and simultaneously proves the feasibility of measuring the micro-nano deformation of the material.
8. The method for measuring the nanoscale three-dimensional deformation under the microscope based on the optical flow algorithm realizes the detection of the out-of-plane deformation of the optical fiber image transmission element under the industrial environment, and further researches show that the detection accuracy is higher when the in-plane displacement of each pixel point of two images before and after deformation is less than 3 pixels.
The foregoing description is only an overview of the technical solutions of the present invention, and in order to make the technical solutions of the present invention more clearly understood and to implement them in accordance with the contents of the description, the following detailed description is given with reference to the preferred embodiments of the present invention and the accompanying drawings.
Drawings
FIG. 1 is a flow chart of a method for measuring nanoscale three-dimensional deformation under a microscope based on an optical flow algorithm according to an embodiment of the present invention;
FIG. 2 shows a schematic diagram of the in-plane displacement proposed by one embodiment of the present invention;
FIG. 3 shows a schematic diagram of the out-of-plane displacement proposed by one embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating the determination of an out-of-plane displacement symbol according to an embodiment of the present invention;
FIG. 5a is a schematic representation of the microscopic expansion deformation proposed by one embodiment of the present invention;
FIG. 5b shows a schematic representation of the microscopically contracted deformation set forth in one embodiment of the present invention;
FIG. 6 is a schematic diagram illustrating a principle of measurement of a wedge mathematical relationship of an in-plane displacement field extraction out-of-plane displacement field according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a solid material three-dimensional deformation measuring apparatus according to an embodiment of the present invention;
FIG. 8a shows a microscope photograph of a material according to an embodiment of the present invention before deformation;
FIG. 8b shows a microscope photograph of a material after deformation as proposed by an embodiment of the present invention;
FIG. 9a is a graph showing in-plane displacement measurements in the x-direction for a material proposed by an embodiment of the present invention;
FIG. 9b shows a graph of in-plane displacement measurements of materials in the y-direction as proposed by an embodiment of the present invention;
fig. 9c shows a graph of the out-of-plane displacement measurements of the material in the z-direction as proposed by an embodiment of the present invention.
Detailed Description
To further illustrate the technical means and effects of the present invention for achieving the predetermined objects, the following detailed description of the method for measuring three-dimensional deformation of a microscope based on an optical flow algorithm, its specific implementation, structure, features and effects will be given with reference to the accompanying drawings and the preferred embodiments. In the following description, different "one embodiment" or "an embodiment" refers to not necessarily the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
As shown in fig. 1, a method for measuring a nanoscale three-dimensional deformation under a microscope based on an optical flow algorithm provided by an embodiment of the present invention specifically includes the following steps:
s101, respectively obtaining an orthographic projection image before material deformation and an orthographic projection image after deformation; the specific operation is as follows: the method comprises the following steps of coaxially placing and packaging a light source, a three-dimensional light-transmitting object stage, a microscope and an industrial camera in a dark box, starting the light source, placing a solid material to be measured on the light-transmitting object stage above the light source, and adjusting the magnification factor of the microscope and the focal length of the industrial camera to enable an image of the upper surface of the solid material to be clearly shot by the industrial camera; the method comprises the steps of shooting an orthographic two-dimensional image of a solid material before deformation, then setting a shooting time interval according to the deformation speed, and shooting the orthographic two-dimensional image of the solid material after deformation.
In step S101, since the optical flow algorithm is sensitive to the change of the light intensity, the measuring device needs to be enclosed in a closed dark box for measurement. The center lines of the light source, the solid material, the microscope and the industrial camera are collinear. As shown in fig. 7, the measuring apparatus of this step is shown, wherein 1 is an optical dark box, 2 is a computer, 3 is an industrial camera, 4 is an eyepiece of a microscope, 5 is an objective of the microscope, 6 is a solid material to be measured, 7 is a three-dimensional light-transmitting stage, and 8 is a light source. The specific operation process is as follows: the light source 8 is turned on, and the position of the three-dimensional light-transmitting stage 7, the magnification of the objective lens 5 of the microscope and the focal length of the industrial camera 3 are adjusted, so that the image of the upper surface of the solid material 6 to be measured can be clearly photographed by the industrial camera.
In this step, in order to ensure image quality and alignment, the microscope used is a pure optical magnifying microscope, and images are not obtained in an electron magnifying manner. The actual out-of-plane displacement height may be obtained by converting the unit of out-of-plane displacement from pixel units to length units according to the magnification of the microscope.
And setting a shooting time interval (namely a time factor) according to the deformation speed, obtaining two images before and after deformation, performing a pretest on the out-of-plane displacement fields of the two images by using a horns-Schunck optical flow method, and multiplying the out-of-plane motion field by the time factor to obtain the out-of-plane displacement field.
Further, to ensure the calculation time and the measurement accuracy, the size of the two images before and after the deformation of the material is set between 256 × 256 pixels and 1024 × 1024 pixels, preferably between 256 × 256 pixels and 512 × 512 pixels.
S102, obtaining an in-plane displacement field between two images before and after deformation by using an optical flow algorithm;
further preferably, the optical flow algorithm is a Horn-Schunck optical flow algorithm.
And (3) performing pretest on the in-plane motion fields of the two images by using a horns-Schunck optical flow algorithm, ensuring that the maximum value of the in-plane displacement of all points between the two images is less than 3 pixels, otherwise, reducing the set time interval until the maximum value of the in-plane displacement of all points between the two images is less than 3 pixels, and then starting the subsequent steps. Two plots within this range give better measurements.
The optical flow algorithm is suitable for measuring micro deformation, and a better result can be obtained when the in-plane displacement is within 3 pixels, so that the method needs to perform pre-measurement and select a proper time factor, the measurement of the out-of-plane displacement can be continued when the maximum value of the in-plane displacement is less than 3 pixels, and a smaller time factor is selected otherwise. Therefore, the measurement precision of the method can reach below 100nm, and the method can be used for detecting the nanoscale deformation.
S103, obtaining an out-of-plane displacement field of material deformation from the in-plane displacement field according to the mathematical relation of the wedge surfaces;
in step S103, an in-plane displacement field between the two images is obtained by using an optical flow algorithm, and since the deformation of the object observed under the microscope can be regarded as the wedge surface deformation, an out-of-plane displacement field (out-of-plane displacement scalar field) of the object can be obtained from the in-plane displacement field of the object according to the mathematical relation of the wedge surface deformation.
The measurement principle of the method is as follows:
1. in-plane Displacement (In-plane Displacement): when an object moves on a two-dimensional plane (xy plane), vectors of movement of each pixel point in the x direction and the y direction are generally called u and v as in-plane displacement in the x direction and y direction, as shown in fig. 2.
2. Out-of-plane displacement (Out-plane displacement): if the object is deformed in the z-axis direction (which can be considered as being raised or lowered by a height value), the displacement in the z-axis direction is called out-of-plane displacement, and its direction is two, i.e. upward and downward, and the upward direction is shown in fig. 3.
3. Judgment principle of out-of-plane displacement vector field (direction): the solid material is deformed for many reasons, such as heat, cold, high pressure, etc. When deformation occurs, the material is usually convex (up) or concave (down) overall, and the center of the convex or concave is usually at the central position of the material, namely the deformation occurrence source point. Let the in-plane displacement field in the x-direction be u and the in-plane displacement field in the y-direction be v, and if a rectangular coordinate system is established with the deformation generation source point as the center of the circle, the directions of the in-plane displacement fields u and v are determined by the quadrant position where the observation point is located and the direction of the out-of-plane displacement, as shown in fig. 4. Thus, the direction of the out-of-plane displacement, i.e. the out-of-plane displacement vector field, can be determined by the direction of the in-plane displacement fields u, v and the quadrant in which the viewpoint is located. Or according to the principle of expansion with heat and contraction with cold, it can be judged that the out-of-plane displacement generated by the material expanded by heat is convex, namely, expansion type deformation, as shown in fig. 5 a; the direction of the out-of-plane displacement produced upon cold contraction is concave, i.e. a contracting type deformation, as shown in fig. 5 b.
4. Optical flow algorithm principle: and setting the gray value at the pixel point (x, y) at the moment t as I (x, y, t), moving the point to a new position (x + delta x, y + delta y) at the moment (t + delta t), and marking the gray value as I (x + delta x, y + delta y). According to the assumption of image consistency, namely that the brightness of the image along the motion track is kept constant, the method meets the requirement
To give formula (5):
I(x,y,t)＝I(x+Δx,y+Δy,t+Δt) (5)
let u and v be the two components of the light flow vector at the point along the x and y directions, respectively, and
unfolding I (x + Δ x, y + Δ y, t + Δ t) with Taylor to give formula (6):
neglecting the higher order terms of the order of two or more in equation (6) yields equation (7):
transforming formula (7) to (8):
due to Δ t → 0, formula (9) is obtained:
Ixu+Iyv+It＝0 (9)
wherein the content of the first and second substances,
Ix、Iyand ItThe partial derivatives of the image I with respect to x, y and t are obtained, and the values of the partial derivatives can be estimated by using the first-order difference of the target pixel points of the adjacent images in the image sequence. Equation (9) is the basic equation for optical flow.
Since equation (9) has only one equation, only the value I of the light flow in the gradient direction can be determinedx、IyAnd ItAnd (u, v) is the optical flow to be solved, and the solution of u and v is not unique, so that a new constraint condition needs to be added to solve the velocity vector.
The calculation of the optical flow field can be classified into a Horn-Schunck algorithm, a Lucas-Kanade algorithm, a Brox algorithm, and the like according to constraints.
The basic solving process of the optical flow algorithm is described by taking the Horn-Schunck algorithm as an example. The basic idea of the Horn-Schunck algorithm is to require the optical flow itself to be as smooth as possible when solving the optical flow. By smoothing, it is within a given neighborhood
As small as possible, formula (10) is obtained:
combining equation (9) and equation (10), the Horn-Schunck algorithm resolves the computation of optical flow (u, v) to the variational problem of equation (11).
Corresponding Euler-Lagrange equations can be obtained, and the solution is carried out by using a Gauss-Seidel method to obtain (n +1) th iterative estimation (u) of each position on the imagen+1,vn+1) Comprises the following steps:
wherein the content of the first and second substances,
and
is the average of u and v in the sub-region centered on (x, y), and α is the smoothing parameter. U obtained by iteration according to equations (3) and (4)n+1And vn+1I.e. the two components u and v of the in-plane displacement field between the two frame images.
5. The principle of extracting the out-of-plane displacement field by the in-plane displacement field (the wedge surface mathematical relationship): the deformation is regarded as a wedge-surface deformation, i.e. a small wedge angle θ of the lifted object, as shown in fig. 6, where OA is OA ═ OA1When viewed from the vertical direction, the point a moves to a ', the out-of-plane displacement generated in the vertical direction is w, the total in-plane displacement before and after the deformation is d to AA', and R is the distance from the point a to the fulcrum O on the image.
Regarding the deformation as wedge surface deformation, according to the mathematical relationship of the wedge surfaces, the out-of-plane displacement w of the material deformation is calculated by the formula (1):
w＝R sinθ (1)
in formula (1), R is the image width; theta is a deformation wedge angle calculated by formula (2), and the deformation wedge angle can be obtained by deformation of formula (2)
In the formula (2), d is an in-plane displacement field between two images before and after changing; r is the image width and is known; theta is a deformation wedge angle; u and v are two components of an in-plane displacement field between two images before and after deformation respectively, and are obtained by utilizing the Horn-Schunck optical flow algorithm for calculation, and u is obtained by iteration of an equation (3) and an equation (4)n+1And vn+1I.e., u and v:
in the formulae (3) and (4),
and
the average values of u and v in the sub-region centered on (x, y), respectively, and α is a smoothing parameter.
6. Working principle diagram: firstly, an in-plane displacement field between two images is obtained by using a Horn-Schunck optical flow algorithm, and since the deformation of an object observed under a microscope can be regarded as wedge surface deformation, an out-of-plane displacement field (an out-of-plane displacement scalar field) of the object can be obtained from the in-plane displacement field of the object according to a mathematical relation of the wedge surface deformation. The measurement device shown in fig. 6 is adopted, a shooting time interval (i.e. time factor) is set according to the deformation speed, two images before and after deformation are obtained, a Horn-Schunck optical flow algorithm is used for pretesting the in-range motion field of the two images, the in-range motion field is multiplied by the time factor to obtain an in-range displacement field, it is noted that the in-range displacement field of the two images needs to be pretested first, the maximum value of the in-range displacement of all points between the two images is ensured to be less than 3 pixels, otherwise, the set time interval needs to be reduced until the maximum value of the in-range displacement of all points between the two images is less than 3 pixels, and then the subsequent steps are started.
The invention provides a method for measuring nanoscale three-dimensional deformation under a microscope based on an optical flow algorithm, which further comprises the following steps:
and S104, establishing a rectangular coordinate system by taking the position of the deformation generating source as an origin, judging whether the direction of the out-of-plane motion vector field is upward or downward through a quadrant where a measuring point (an observation position) is located, a horizontal component symbol and a vertical component symbol of the in-plane displacement field, and finally multiplying the three-dimensional displacement field by a time factor respectively to obtain the deformed in-plane and out-of-plane displacement fields. FIG. 4 is a schematic diagram illustrating the judgment of the out-of-plane displacement symbol.
In step S104, the vector direction of the out-of-plane displacement may also be determined by macroscopically observing whether the deformation characteristic of the material is expansion type or contraction type, where the out-of-plane displacement vector direction of the expansion deformation is inclined-surface convex, and the out-of-plane displacement vector direction of the contraction deformation is inclined-surface concave. The vector direction (vector field) of the out-of-plane displacement can also be determined a priori: the general material can expand and deform under the action of heat or high pressure, and the direction of the out-of-plane displacement is convex; the material can shrink and deform under the action of cooling or corrosion, and the direction of the out-of-plane displacement is concave.
The method for measuring the nanoscale three-dimensional deformation of the solid material under the microscope based on the optical flow realizes the detection of the nanoscale three-dimensional deformation of the solid material under the industrial environment, is suitable for the condition that the displacement range of each pixel point of two images before and after deformation is less than 3 pixels, is specially provided for the detection of the nanoscale three-dimensional deformation of the solid material under the microscope, and can be used for processing data calculation by Matlab software.
The invention discloses a method for measuring nanoscale three-dimensional deformation under a microscope based on optical flow, which mainly comprises the following steps: the solid material which is deformed after being subjected to physical and chemical action is placed on an objective table of a microscope, and an industrial camera is placed above an ocular lens of the microscope to obtain a deformation image of an amplified object under a microscopic condition. When the deformation scale is in a nanometer magnitude, the two-dimensional images before and after deformation meet the optical flow brightness steady assumption and the image gradient steady assumption, and the in-plane displacement field of the object can be directly obtained by adopting an optical flow algorithm. Meanwhile, the deformation of the object observed under the microscope can be regarded as the wedge surface deformation, and the out-of-plane displacement field of the object can be obtained from the in-plane displacement field of the object according to the mathematical relation of the wedge surface deformation. And then, establishing a rectangular coordinate system by taking the position of the deformation generating source as an origin, and judging the direction of the out-of-plane motion vector field by observing the quadrant where the position is located, the symbol of the horizontal component and the symbol of the vertical component of the in-plane displacement field. And finally, multiplying the three-dimensional displacement field by a time factor respectively to obtain the deformed in-plane and out-of-plane displacement fields. The method can finish measurement only by shooting two images in the video by one camera, does not need an interference light path for testing, has strong applicability to testing environment and is suitable for industrial detection. Meanwhile, the method does not need to convert to a frequency domain or perform phase unwrapping operation, can greatly reduce measurement errors, has high measurement precision, and is suitable for dynamic measurement.
The measuring method designed by the invention is specially provided for measuring the out-of-plane displacement of a solid material (such as an optical fiber image transmission material) under a microscope, and the displacement calculation can be processed by Matlab software.
The present invention will be further described with reference to the following specific examples, which should not be construed as limiting the scope of the invention, but rather as providing those skilled in the art with certain insubstantial modifications and adaptations of the invention based on the teachings of the invention set forth herein.
Example 1
A method for measuring nanoscale three-dimensional deformation under a microscope based on optical flow specifically comprises the following steps:
and 4, establishing a rectangular coordinate system by taking the position of the deformation generating source point as an origin, and judging the direction of the out-of-plane motion vector field by observing the quadrant where the position is located, the symbol of the horizontal component and the symbol of the vertical component of the in-plane displacement field, as shown in fig. 4, the direction is a schematic judgment diagram of the out-of-plane displacement symbol.
And (4) analyzing results: the three-dimensional deformation measurement results obtained by the method proposed by the present invention are shown in fig. 9a, 9b and 9c, wherein fig. 9a shows the horizontal in-plane displacement u, fig. 9a shows that the maximum value of the horizontal in-plane displacement is 80nm, fig. 9b shows the vertical in-plane displacement v, fig. 9b shows that the maximum value of the vertical in-plane displacement is 120nm, fig. 9c shows the out-of-plane z-axis displacement w, and fig. 9c shows that the maximum value of the out-of-plane displacement is 2743 nm.
Compared with actual measurement values, the mean error of the off-plane displacement measured by the method is 5.11%, the error of the off-plane displacement below 2000nm is 2.98%, the calculation time is 12.34 seconds, and the detection efficiency is greatly improved.
In the description of the present invention, it should be noted that the terms "upper", "lower", "horizontal", "vertical", and the like indicate orientations or positional relationships based on methods or positional relationships shown in the drawings, and are only for convenience of describing the present invention and simplifying the description, but do not indicate or imply that the referred device or element must have a specific orientation, be constructed in a specific orientation, and be operated, and thus, should not be construed as limiting the present invention.
In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
Although the present invention has been described with reference to a preferred embodiment, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A method for measuring nanoscale three-dimensional deformation under a microscope based on an optical flow algorithm is characterized by comprising the following steps:
respectively acquiring an orthographic projection image before material deformation and an orthographic projection image after deformation;
obtaining an in-plane displacement field between the two images before and after deformation by using an optical flow algorithm;
and obtaining an out-of-plane displacement field of the material deformation from the in-plane displacement field according to the mathematical relation of the wedge surfaces.
2. The method for measuring nanoscale three-dimensional deformation under microscope based on optical flow algorithm according to claim 1, wherein the optical flow algorithm is used to obtain the in-plane displacement field between two images before and after deformation; according to the mathematical relationship of the wedge surfaces, the out-of-plane displacement field of the material deformation is obtained by the in-plane displacement field, and the method specifically comprises the following steps:
regarding the deformation as wedge surface deformation, according to the mathematical relationship of the wedge surfaces, the out-of-plane displacement field w of the material deformation is calculated by the formula (1):
w＝Rsinθ (1)
in formula (1), R is the image width; θ is a deformed wedge angle, which is calculated by equation (2):
in the formula (2), d is an in-plane displacement field between two images before and after changing; r is the image width and is known; theta is a deformation wedge angle; u and v are two components of an in-plane displacement field between two images before and after deformation respectively, wherein the u and v are obtained by calculation through an optical flow algorithm.
3. The method for measuring nanoscale three-dimensional deformation under microscope based on optical flow algorithm according to claim 2, wherein the optical flow algorithm is Horn-Schunck optical flow algorithm, and u and v are obtained by calculation through the Horn-Schunck optical flow algorithm and are obtained by iteration of formula (3) and formula (4) respectivelyn+1And vn+1：
In the formulae (3) and (4),
and
the average values of u and v in the sub-region centered on (x, y), respectively, and α is a smoothing parameter.
4. The method for measuring nanoscale three-dimensional deformation under microscope based on optical flow algorithm according to claim 1, wherein the in-plane displacement of all points between the two images before and after deformation is less than 3 pixels.
5. The method of claim 1 in which the two images are each between 256 x 256 and 512 x 512 in size.
6. The method for measuring nanoscale three-dimensional deformation under microscope based on optical flow algorithm according to claim 1, wherein the obtaining of the in-plane displacement field between the two images before and after deformation by using optical flow algorithm comprises:
firstly, an in-plane motion field between two images before and after deformation is carried out by utilizing an optical flow algorithm, then the obtained in-plane motion field is multiplied by a time factor to obtain an in-plane displacement field of three-dimensional deformation, and then the unit of out-of-plane displacement is converted into a length unit from a pixel unit according to the magnification of a microscope to obtain the out-of-plane displacement size, wherein the time factor is the time interval between the two images before and after deformation.
7. The method for measuring nanoscale three-dimensional deformation under microscope based on optical flow algorithm according to claim 1, wherein the obtaining of the image before deformation of the material and the orthographic projection image after deformation respectively comprises:
the method comprises the following steps of coaxially placing and packaging a light source, a three-dimensional light-transmitting object stage, a microscope and an industrial camera in a dark box, starting the light source, placing a solid material to be measured on the light-transmitting object stage above the light source, and adjusting the magnification factor of the microscope and the focal length of the industrial camera to enable an image of the upper surface of the solid material to be clearly shot by the industrial camera; the orthographic projection image of the solid material before deformation is shot, then the shooting time interval is set according to the deformation speed, and the orthographic projection image of the solid material after deformation is shot.
8. The method for measuring nanoscale three-dimensional deformation under a microscope based on optical flow algorithm according to claim 7, wherein the microscope is a pure optical magnifying microscope.
9. The method for measuring nanoscale three-dimensional deformation under a microscope based on optical flow algorithm according to claim 1, further comprising:
and establishing a rectangular coordinate system by taking the position of the deformation generating source as an original point, and judging the vector direction of the out-of-plane displacement field according to the quadrant where the measuring point is located, the symbol of the horizontal component and the symbol of the vertical component of the in-plane displacement field.
10. The method for measuring microscopic nanoscale three-dimensional deformation based on optical flow algorithm of claim 1, wherein the method for measuring microscopic nanoscale three-dimensional deformation based on optical flow algorithm is processed by Matlab software.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110246918.4A CN113048901B (en) | 2021-03-05 | 2021-03-05 | Method for measuring nanoscale three-dimensional deformation under microscope based on optical flow algorithm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110246918.4A CN113048901B (en) | 2021-03-05 | 2021-03-05 | Method for measuring nanoscale three-dimensional deformation under microscope based on optical flow algorithm |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113048901A true CN113048901A (en) | 2021-06-29 |
| CN113048901B CN113048901B (en) | 2023-06-27 |
Family
ID=76510271
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110246918.4A Active CN113048901B (en) | 2021-03-05 | 2021-03-05 | Method for measuring nanoscale three-dimensional deformation under microscope based on optical flow algorithm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113048901B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US3940608A (en) * | 1974-02-04 | 1976-02-24 | Mechanical Technology Incorporated | Fiber optic displacement measuring apparatus |
| JPS63165726A (en) * | 1986-08-15 | 1988-07-09 | Sumitomo Electric Ind Ltd | Method for measuring structure of optical fiber |
| US5870195A (en) * | 1995-09-29 | 1999-02-09 | Telefonaktiebolaget L M Ericsson | Method and system for determining the distance between optical fibres |
| US20050146708A1 (en) * | 2002-04-11 | 2005-07-07 | Xunqing Shi | Systems and methods for deformation measurement |
| CN102507446A (en) * | 2011-10-24 | 2012-06-20 | 北京航空航天大学 | Detection method for optical angle deviation of light-transmitting glass |
| US20130280752A1 (en) * | 2011-01-06 | 2013-10-24 | The Regents Of The University Of California | Lens-free tomographic imaging devices and methods |
| CN103808278A (en) * | 2014-03-06 | 2014-05-21 | 西安电子科技大学 | Measurement system and measurement method for optical fibre morphology |
| US20190212130A1 (en) * | 2016-09-27 | 2019-07-11 | National Institute Of Advanced Industrial Science And Technology | Three-dimensional shape, displacement, and strain measurement device and method using periodic pattern, and program therefor |
-
2021
- 2021-03-05 CN CN202110246918.4A patent/CN113048901B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US3940608A (en) * | 1974-02-04 | 1976-02-24 | Mechanical Technology Incorporated | Fiber optic displacement measuring apparatus |
| JPS63165726A (en) * | 1986-08-15 | 1988-07-09 | Sumitomo Electric Ind Ltd | Method for measuring structure of optical fiber |
| US5870195A (en) * | 1995-09-29 | 1999-02-09 | Telefonaktiebolaget L M Ericsson | Method and system for determining the distance between optical fibres |
| US20050146708A1 (en) * | 2002-04-11 | 2005-07-07 | Xunqing Shi | Systems and methods for deformation measurement |
| US20130280752A1 (en) * | 2011-01-06 | 2013-10-24 | The Regents Of The University Of California | Lens-free tomographic imaging devices and methods |
| CN102507446A (en) * | 2011-10-24 | 2012-06-20 | 北京航空航天大学 | Detection method for optical angle deviation of light-transmitting glass |
| CN103808278A (en) * | 2014-03-06 | 2014-05-21 | 西安电子科技大学 | Measurement system and measurement method for optical fibre morphology |
| US20190212130A1 (en) * | 2016-09-27 | 2019-07-11 | National Institute Of Advanced Industrial Science And Technology | Three-dimensional shape, displacement, and strain measurement device and method using periodic pattern, and program therefor |
Non-Patent Citations (4)
| Title |
|---|
| 史红健等: "一种三维变形光学测量系统及微结构力学性能测试应用", 《光学学报》 * |
| 孔超等: "基于体视显微镜的测量系统标定研究", 《实验力学》 * |
| 孙平等: "条纹光流法三维变形测量", 《第十七届全国光学测试学术交流会》 * |
| 赵冉: "基于条纹频率的无包络三维变形测量技术研究", 《中国优秀博士学位论文全文数据库（电子期刊）》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113048901B (en) | 2023-06-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110779797B (en) | Method for measuring plastic strain ratio in metal tensile test process | |
| Li et al. | Pose sensing and servo control of the compliant nanopositioners based on microscopic vision | |
| Niu et al. | A new in-situ warpage measurement of a wafer with speckle-free digital image correlation (DIC) method | |
| CN113899698B (en) | Real-time focusing and centering adjustment method and device for in-situ test platform | |
| Wang et al. | Resolution enhancement for topography measurement of high-dynamic-range surfaces via image fusion | |
| Tang et al. | Micro-assembly precise coaxial alignment methodology based on surface roughness and reflectiveness matching | |
| CN112697063B (en) | Chip strain measurement method based on microscopic vision | |
| Zhang et al. | High-accuracy three-dimensional shape measurement of micro solder paste and printed circuits based on digital image correlation | |
| CN113048901A (en) | Method for measuring nanoscale three-dimensional deformation under microscope based on optical flow algorithm | |
| Benabou et al. | Development and first assessment of a DIC system for a micro-tensile tester used for solder characterization | |
| CN110514110B (en) | Platform leveling control method | |
| Kang et al. | New measurement method of Poisson’s ratio of thin films by applying digital image correlation technique | |
| Li et al. | Triangulation-based edge measurement using polyview optics | |
| Chen et al. | Micro-vision servo control of a multi-axis alignment system for optical fiber assembly | |
| Wang et al. | Multi-depth-of-field 3-D profilometry for a microscopic system with telecentric lens | |
| CN106556350B (en) | The measuring method and a kind of microscope of microscopic slide curved surface height value | |
| CN113063356B (en) | Method for measuring directional deformation out-of-plane displacement of material | |
| Danuser | Stereo light microscope calibration for 3D submicron vision | |
| Malti et al. | Magnification-continuous static calibration model of a scanning-electron microscope | |
| Hartmann et al. | Digital image correlation and optical flow analysis based on the material texture with application on high-speed deformation measurement in shear cutting | |
| Sievers et al. | Vision feedback in an automatic nanohandling station inside an SEM | |
| Kim et al. | A novel depth-from-focus-based measurement system for the reconstruction of surface morphology with depth discontinuity | |
| Furlong et al. | New optoelectronic methodology for nondestructive evaluation of MEMS at the wafer level | |
| CN113409311B (en) | NSST-based contact element distribution and morphology information fusion method | |
| Kim et al. | Iterative algorithm for automatic alignment by object transformation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |