CN116758160A - Position and orientation detection method and assembly method of optical components assembly process based on orthogonal vision system - Google Patents

Abstract

The position and orientation detection method and assembly method of optical component assembly process based on orthogonal vision system belong to the field of industrial assembly visual detection. The present invention solves the problem in the existing optical component assembly that the edge of the optical component and the assembly frame are prone to collision, resulting in assembly errors. . The method of the present invention includes: S1, the step of constructing a visual detection system; the visual detection system includes one global camera and two side-view cameras; S2, the joint calibration step of the three cameras in the visual detection system; wherein, the global camera The camera coordinate system is used as a unified global coordinate system; S3, the optical element pose solution step; the three cameras simultaneously obtain the top view and two side views of the optical element, and extract the respective pixel coordinates of the edge of the optical element from the images of the three cameras The analytical formula under the system, combined with the joint calibration data, aligns the edges in the pixel coordinate systems of different cameras to a unified global coordinate system; the pose of the optical element is determined through the analytical formula of these edges.

Description

Position detection method and assembly of optical components during assembly based on orthogonal vision system method

Technical field

The invention relates to the field of visual inspection of industrial assembly, specifically a method for solving the six-degree-of-freedom pose of optical components in the assembly of industrial automated optical components.

Background technique

In the process of automated assembly of optical components, a robot is used to hold the optical components and place them in the mechanical frame to realize automated assembly of optical components. During the assembly process, high accuracy is required for detecting the relative positions and postures between optical components, manipulators, and assembly frames. The existing direct visual inspection method has poor positioning reliability and limited accuracy for transparent optical components. In addition, there are non-negligible manufacturing tolerances in both optical components and assembly frames. Therefore, the optical components are assembled on the FOA assembly platform (final optics assembly, terminal optical system). The edge of the optical element is prone to collide with the assembly frame, leading to assembly errors.

Contents of the invention

In order to solve the problem of easy collision between the edge of the optical element and the assembly frame in the existing optical element assembly, resulting in assembly errors, the present invention provides a visual inspection method and assembly method for optical element assembly based on an orthogonal visual inspection system. Deploy vision to detect the edge position of the optical element, and combine it with the joint calibration data to calculate the geometric size, spatial position, and attitude of the optical element, achieve reliable and precise positioning of the optical element, and guide the manipulator to adjust its posture.

The pose detection method of the optical element assembly process based on the orthogonal vision system of the present invention includes the following steps:

S1. Steps to build a visual inspection system;

The visual inspection system includes a global camera and two side-view cameras, in which a strip light source is used in front of the two side-view cameras to illuminate the optical elements glancingly;

S2. Joint calibration steps of three cameras in the visual inspection system;

Among them, the camera coordinate system of the global camera is used as the unified global coordinate system; the internal and external parameters of the three cameras are calibrated, and the spatial mapping relationship between the pixel coordinate system of each camera and the unified global coordinate system is established;

S3. Optical component pose calculation steps;

Three cameras simultaneously acquire the top view and two side views of the optical element. The analytical formulas of the edge of the optical element in their respective pixel coordinate systems are extracted from the images of the three cameras. Combined with the joint calibration data, the pixel coordinates of the different cameras are combined. The edges are aligned to a unified global coordinate system; the pose of the optical element is determined through the analytical expressions of these edges.

Preferably, the optical element to be inspected is placed on the FOA assembly platform in the horizontal direction, and the global camera is set above the FOA assembly platform. The two cameras located on the side of the optical element during inspection are Side View Camera No. 1 and Side View Camera No. 2.

The postures of the three cameras are orthogonal to each other, and the optical axes of the three cameras intersect at a point on the upper surface of the optical element. The following coordinate system is established:

The global coordinate system of the global camera is O _G -X _G Y _G Z _G , and the optical axis direction of the global camera is perpendicular to the horizontal plane;

The coordinate system of side view camera No. 1 is O _C1 -X _C1 Y _C1 Z _C1 , and the coordinate system of side view camera No. 2 is O _C2 -X _C2 Y _C2 Z _C2 . The optical axes of the two cameras are parallel to the horizontal plane;

The positive direction of the X axis of the global camera coordinate system O _G is the same as the positive direction of the Z _C2 axis of the No. 2 side view camera coordinate system O _C2 ;

The model coordinate system O _M of the optical element is O _M -X _M Y _M Z _M , and the origin is defined at the vertex A in the upper left corner of the upper surface of the optical element. The Z _M axis is perpendicular to the upper surface of the optical element, and the positive direction of the Z _M axis is upward. , the _positive direction of the

Preferably, the joint calibration process of step S2 is:

First, the Zhang Zhengyou calibration method is used to calibrate the internal parameter matrices of the three cameras respectively, and establish the mapping relationship between the respective two-dimensional pixel coordinate system and the three-dimensional camera coordinate system;

Then use the camera coordinate system of the global camera as the unified global coordinate system, respectively calibrate the coordinate system transformation from the camera coordinate system of the two side-view cameras to the global coordinate system, and obtain the extrinsic parameter matrices of the two side-view cameras;

The external parameter matrix includes: the conversion relationship T _C1 = ( _RC1 , t _C1 ) from the No. 1 side view camera coordinate system O _C1 to the global coordinate system O _G. R _C1 is the No. 1 side view camera coordinate system O _C1 to the global coordinate system O G. The rotation matrix of the coordinate system O _G , t _C1 is the translation vector from the No. 1 side view camera coordinate system O _C1 to the global coordinate system O _G ;

And the conversion relationship between the No. 2 side-view camera coordinate system O _C2 and the global coordinate system O _G is T _C2 = ( _RC2 ,t _C2 ). R _C2 is the conversion relationship between the No. 2 side-view camera coordinate system O _C2 and the global coordinate system O _G. Rotation matrix, t _C2 is the translation vector from the No. 2 side view camera coordinate system O _C2 to the global coordinate system O _G.

Preferably, a stereo calibration block is used in combination with the AprilTag algorithm to calibrate the extrinsic parameter matrices of the two side view cameras. The stereo calibration block is a cuboid, and the four sides and the top surface are sequentially defined as the surface 0-surface 4 of the cube, and the surface 0- Surface 4 adopts the 36H11 AprilTag pattern with IDs 0-4 in turn. The origin of the coordinate system O _B of the three-dimensional calibration block is defined at the center of the AprilTag pattern (id=0) on surface 0. The X-axis and Y-axis are parallel to the surface, and the positive direction They are the direction from the upper left vertex of the pattern to the upper right vertex, the direction from the upper left vertex of the pattern to the lower left vertex, and the Z-axis direction points to the inside of the model;

The process of calibrating the external parameter matrix using the AprilTag algorithm is:

Keep the surface 4 of the three-dimensional calibration block horizontal. Select one side from the surfaces 0-3 of the three-dimensional calibration block. Let the distance between the AprilTag pattern on the selected surface and the optical center of the No. 1 side-view camera along the optical axis direction be 280mm. The AprilTag pattern on the surface 4 The distance to the optical center of the global camera along the optical axis is 1200mm, and the corresponding AprilTag pattern must appear completely in the fields of view of the above two cameras;

When the stereo calibration block, the global camera, and the No. 1 side view camera meet the above requirements, use the external trigger method to control the two cameras to capture images with a time difference of no more than 15ms, and use the AprilTag algorithm to calculate the positions of the calibration graphics in the field of view to the respective cameras. Attitude; the transformation relationship between the coordinate system of all graphics on the stereo calibration block and the coordinate system O _B of the calibration block is known from the CAD data. The pose of the calibration block relative to the global camera and the No. 1 side view camera at the time of taking the photo is deduced. Solution Calculate the transformation between these two poses T _C1 = (R _C1 ,t _C1 ), which is the conversion relationship from O _C1 to O _G , and obtain the external parameter matrix of side view camera No. 1 under O _G ;

The extrinsic parameter matrix of No. 2 side-view camera under O _G is obtained in the same way as the above process.

Preferably, the optical element pose solution process in step S3 is:

S31. Use the fixture to place the optical element at the predetermined position on the FOA platform, let the three cameras simultaneously collect the top view image I _M and the two side view images I _SL and I _SF of the optical element, and preprocess the three images to generate a grayscale image. ;

S32. Use the LSD algorithm (Line Segment Detector, line segment detector algorithm) to extract line segments from the three grayscale images, process these line segments, and obtain the edges of the optical elements of the three grayscale images in the respective camera pixel coordinate systems;

S33. Obtain the analytical formula of the edge of the optical element in the three camera coordinate systems. Specifically, according to the matching result of step S32, when an edge in the top-view image _IM matches an edge in the side-view image I _SL or the side-view image I _SF If a certain edge simultaneously corresponds to the same edge in the CAD model, it is considered that the edges in the above two images correspond to the same edge of the optical element; according to this principle, the analytical formulas of the three grayscale images in their respective camera coordinate systems are obtained;

S34. According to the camera internal parameter matrix, external parameter matrix, and the analytical formulas in the three camera coordinate systems, obtain the analytical formula of the edge of the optical element in the global coordinate system;

S35. According to the analytical formula of the edge of the optical element in the global coordinate system, use the PnP algorithm to calculate the position, posture and size of the upper surface of the optical element based on the corresponding relationship.

Preferably, the acquisition process of the edge of the optical element in step S32 is:

The LSD algorithm is used to extract all line segments from the three grayscale images and processed as follows:

Step 1. Eliminate too short line segments with L<25 pixels. The length of the line segment L is obtained as follows:

(x ₁ , y ₁ ), (x ₂ , y ₂ ) are the pixel coordinates of the two end points of the line segment respectively;

Step 2: Record the set of unfused line segments as l _Ori and the set of fused line segments as l _meg . The remaining line segments after eliminating the short line segments in step 1 are first placed in the set l _Ori ;

Step 3. Fusion of line segments:

Search for the line segment with the largest L in the set l _Ori , recorded as l ₀ , and search for the line segment that meets the conditions in the set l _Ori : the Euclidean distance dis between the midpoint of the line segment and the straight line where l ₀ is less than 3 pixels, and the angle between the line segment and l ₀ ang is less than 0.5°; remove l ₀ and qualified line segments from the set l _Ori ;

The Euclidean distance dis and angle ang between two line segments are calculated as follows:

in:

(a ₁ ,b ₁ ), (a ₂ ,b ₂ ) are the endpoint coordinates of l ₀ ,

k is the slope of l ₀ ,

x＝(x ₁ +x ₂ )/2

y＝(y ₁ +y ₂ )/2

Δx＝x ₂ -x ₁

Δy＝y ₂ -y ₁

Δa＝a ₂ -a ₁

Δb＝b ₂ -b1

Step 4. For l ₀ and qualified line segments, use the SMBR algorithm to calculate the minimum circumscribed rectangle of the endpoints of these line segments, and use the pixel coordinates of the midpoint of the two short sides of the circumscribed rectangle as the pixel coordinates of the two endpoints of the new fused line segment. Put this fused line segment into the fused line segment set l _meg ;

Repeat steps three and four until the set l _Ori is an empty set, and then perform step five;

Step 5. Delete the wrong line segments in the set l _meg :

First, delete line segments with L<1200 pixels;

Secondly, the prior knowledge of the optical element CAD model is used to identify the line segments belonging to the edge of the optical element. Specifically, the optical element clamped by the fixture is reset to a predetermined position, and the CAD of the optical element at this time is projected into the collected image. Since the extracted line segments come from the edge of the optical element, the line segments representing the edge of the optical element are filtered according to the following conditions: the Euclidean distance dis between the midpoint of the line segment and the edge line segment of a certain model extracted by CAD is less than 15 pixels, and the included angle ang is less than 2° ; Delete the line segments that do not meet the above conditions in the set l _meg , and use the remaining line segments in the set l _meg as the edge of the optical element.

Preferably, the process of obtaining the analytical expression of the edge of the optical element in the global coordinate system in S34 is:

According to the known internal parameter matrix and external parameter matrix of the camera, the straight line corresponding to the same edge of the optical element in the two images is converted to the plane under the camera coordinate system using back projection, and the three-dimensional parameters of the plane where the straight line is located are obtained. The plane passes through the camera light. Center O; the two images are a top view and a side view;

Assume that the edge line segment l ₁ from the global camera and the edge line segment l ₂ of a side view camera correspond to the same straight line in space, l ₁ and the optical center of the global camera determine the plane P ₁ , l ₂ and the optical center of the side view camera determine For plane P ₂ , planes P ₁ and P ₂ are transformed into the global coordinate system according to the internal parameter matrix and external parameter matrix, and the analytical formulas of P ₁ and P ₂ in the global coordinate system are obtained:

P ₁ :a ₁ X+b ₁ Y+c ₁ Z+d ₁ =0

P ₂ :a ₂ X+b ₂ Y+c ₂ Z+d ₂ =0

Among them, the vector (a ₁ , b ₁ , c ₁ ) is the normal vector of the plane P ₁ , d ₁ is the suffix that satisfies all points on the plane P ₁ ; the vector (a ₂ , b ₂ , c ₂ ) is the normal vector of the plane P ₂ Vector, d ₂ is the suffix that satisfies all points on the plane P ₂ ; c ₁ =c ₂ =1;

Since in the imaging of two cameras, the matching straight line corresponds to the same straight line in the three-dimensional space, the planes P ₁ and P ₂ must intersect, and the intersection line is the target straight line, that is, the analytical formula of the straight line where the edge of the optical element is located is:

P＝p ₀ +tV

Where p ₀ = (X ₀ , Y ₀ , Z ₀ ) is a known point on the target straight line, t is the unit direction vector of the target straight line, in the form of t=iX+jY+kZ, coefficients i, j, k Satisfy the relationship: i ² +j ² +k ² =1;

Repeating this method can obtain the analytical formula of the straight line in the global coordinate system where the upper surface of the optical element is close to the two adjacent edges of the two side-view cameras:

L ₁ :P＝p ₁ + ₁ V

L ₂ :P＝p ₂ + ₂ V

Among them, p ₁ is a known point on the straight line L ₁ , t ₁ is the unit direction vector of the straight line L ₁ , p ₂ is a known point on the straight line L ₂ , t ₂ is the unit direction vector of the straight line L ₂ ;

Calculate the point _A = (X ^M , Y ^M _, Z ^M ) where the sum of the distances to L ₁ and L ₂ is the smallest Points B and C are 410mm apart, and a plane is determined by A, B, and C:

P ₀ :a ₀ X+b ₀ Y+Z+c ₀ =0

That is the analytical formula of the upper surface of the optical element in the global coordinate system. According to the above method, the intersection line between the upper surface and the plane where the other two edges are located is solved, which is the spatial analytical formula of the straight line where the remaining two edges are located in the global coordinate system. :

L ₃ : P＝p ₃ + ₃ V

L ₄ : P＝p ₄ + ₄ V

Among them, p ₃ is a known point on the straight line L ₃ , t ₃ is the unit direction vector of the straight line L ₃ , p ₄ is a known point on the straight line L ₄ , and _{t t} is the unit direction vector of the straight line L ₄ .

Preferably, the process of S35 to calculate the position and posture of the optical element and the size of the upper surface is as follows:

The positive Z-axis direction of the attitude from the optical element model coordinate system O _M to the global coordinate system O _G is defined as t ₂ ×t ₁ , and the positive X-axis direction is defined as t ₂ . Construct PnP correspondence:

The two-dimensional point (0,0) corresponds to the three-dimensional point A=(X _M , Y _M , Z _M );

The two-dimensional point (1,0) corresponds to the three-dimensional point A+t ₂ ;

Two-dimensional point (0,1) corresponds to three-dimensional point A+Norm(t ₂ ×t ₁ )×t ₂ ;

Two-dimensional point (1,1) corresponds to three-dimensional point A+Norm(t ₂ ×t ₁ )×t ₂ +t ₂ ;

Among them, Norm(t) is the operation of normalizing t;

By establishing the PnP relationship of the above four groups of points, the conversion relationship (R, t) from the model coordinate system O _M of the optical element to the global coordinate system O _G can be solved. (R, t) is used as the 6-degree-of-freedom position of the optical element. pose, R is the rotation matrix from the model coordinate system O _M of the optical element to the global coordinate system O _G , t is the translation vector from the model coordinate system O _M of the optical element to the global coordinate system O _G ;

Position matrix t=A ^T =(X _M , Y _M , Z _M ) ^T ;

According to the method of solving the coordinates of point A in the global coordinate system, obtain the coordinates of the remaining vertices on the upper surface of the optical element in the global coordinate system. The remaining vertices are B, C, and D in clockwise order.

The length of the optical element is L=||((A+B)-(D+C))/2|| ₂ and the width is W=||((A+D)-(B+C))/2| | ₂ .

The present invention also provides another technical solution: an optical element assembly method based on an orthogonal vision system. During assembly, the posture detection method of the optical element assembly process based on the orthogonal vision system described in claims 1-8 is used to obtain real-time The pose of the optical element is used to calculate the deviation between the current pose of the optical element and the ideal pose, and guide the manipulator to adjust the pose.

Beneficial effects of the present invention:

(1) High-precision positioning: Through a visual inspection system composed of three cameras, using grazing light source illumination, and clearly imaging the edges of optical elements in a targeted manner, the positioning accuracy of transparent optical elements is effectively improved.

(2) Reduce the impact of manufacturing tolerances: By recalculating the length and width of the upper surface of the optical element, the impact of manufacturing tolerances on the assembly process is reduced.

(3) Accurately measure the six-degree-of-freedom posture: Using computer vision algorithms and based on the images obtained by three cameras, the six-degree-of-freedom posture of the optical element is accurately measured to provide path guidance for the subsequent assembly process.

Description of the drawings

Figure 1 is a schematic structural diagram of the pose detection method in the assembly process of optical components based on an orthogonal vision system according to the present invention;

Figure 2 is a schematic diagram of the three-dimensional calibration block, where Figure 2(a) is a schematic diagram of the surface 4, Figure 2(b) is a schematic diagram of the three-dimensional structure of the three-dimensional calibration block, Figure 2(c) is a schematic diagram of the surface 3 of the three-dimensional calibration block, Figure 2(d) ) is a schematic diagram of the surface 0 of the three-dimensional calibration block, Figure 2(e) is a schematic diagram of the surface 1 of the three-dimensional calibration block, Figure 2(f) is a schematic diagram of the surface 2 of the three-dimensional calibration block,;

Figure 3 is a schematic diagram of back-projection of line segments in three-dimensional space.

Description of each component in the attached figure: 1. Global camera, 2. Side view camera No. 1, 3. Side view camera No. 2, 4. FOA assembly platform, 5. Optical components, 6. Top view, 7. Side view camera No. 1 Side view taken, side view taken by side view cameras No. 8 and 2.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without any creative work fall within the scope of protection of the present invention.

It should be noted that, as long as there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but shall not be used as a limitation of the present invention.

Specific Embodiment 1: This embodiment will be described below with reference to Figures 1 to 3. The posture detection method of the optical element assembly process based on the orthogonal vision system described in this embodiment includes the following steps:

S1. Steps to build a visual inspection system;

The visual inspection system includes a global camera and two side-view cameras, in which a strip light source is used in front of the two side-view cameras to illuminate the optical elements glancingly;

S2. Joint calibration steps of three cameras in the visual inspection system;

Among them, the camera coordinate system of the global camera is used as the unified global coordinate system; the internal and external parameters of the three cameras are calibrated, and the spatial mapping relationship between the pixel coordinate system of each camera and the unified global coordinate system is established;

S3. Optical component pose calculation steps;

Three cameras simultaneously acquire the top view and two side views of the optical element. The analytical formulas of the edge of the optical element in their respective pixel coordinate systems are extracted from the images of the three cameras. Combined with the joint calibration data, the pixel coordinates of the different cameras are combined. The edges are aligned to a unified global coordinate system; the pose of the optical element is determined through the analytical expressions of these edges.

Regarding step S1, see Figure 1. The method of this embodiment is implemented based on an orthogonal vision system. In order to clearly image the optical elements from different angles at the same time and provide image information for subsequent optical element pose detection, the visual inspection system is designed to consist of three cameras. , including a global camera and two side-view cameras, in which a strip light source is used in front of the two side-view cameras to illuminate the optical elements.

The optical element to be inspected is placed on the FOA assembly platform in the horizontal direction, and the global camera is set above the FOA assembly platform. The two cameras located on the side of the optical element during inspection are the No. 1 side view camera and the No. 2 side view camera. The global camera is configured The working distance of No. 1 Side View Camera and No. 2 Side View Camera is 280mm.

The postures of the three cameras are orthogonal to each other, and the optical axes of the three cameras intersect at a point on the upper surface of the optical element. The following coordinate system is established:

The global coordinate system of the global camera is O _G -X _G Y _G Z _G , and the optical axis direction of the global camera is perpendicular to the horizontal plane;

The coordinate system of side view camera No. 1 is O _C1 -X _C1 Y _C1 Z _C1 , and the coordinate system of side view camera No. 2 is O _C2 -X _C2 Y _C2 Z _C2 . The optical axes of the two cameras are parallel to the horizontal plane;

The positive direction of the X axis of the global camera coordinate system O _G is the same as the positive direction of the Z _C2 axis of the No. 2 side view camera coordinate system O _C2 ;

The model coordinate system O _M of the optical element is O _M -X _M Y _M Z _M , and the origin is defined at the vertex A in the upper left corner of the upper surface of the optical element. The Z _M axis is perpendicular to the upper surface of the optical element, and the positive direction of the Z _M axis is upward. , the _positive direction of the

Regarding the joint calibration of step S2, in order to calibrate the internal and external parameters of the three cameras of the visual inspection system and establish the spatial mapping relationship between their respective pixel coordinate systems and the unified global three-dimensional coordinate system, first calibrate the internal parameters of the three cameras separately and establish The spatial mapping relationship between the pixel coordinate system and the respective camera coordinate system, and then the camera coordinate system of the global camera is defined as a unified global coordinate system, and the coordinate system transformation from the camera coordinate system of the two side-view cameras to the global coordinate system is calibrated respectively. That is, the external parameters of the two cameras.

The internal parameter matrix adopts Zhang Zhengyou's calibration method to calibrate the internal parameter matrix of the three cameras and establish the mapping relationship between the respective two-dimensional pixel coordinate system and the three-dimensional camera coordinate system;

Among them, the external parameter matrix calibration uses the camera coordinate system of the global camera as a unified global coordinate system, respectively calibrates the coordinate system transformation from the camera coordinate system of the two side-view cameras to the global coordinate system, and obtains the external parameter matrices of the two side-view cameras. ;

The external parameter matrix includes: the conversion relationship T _C1 = ( _RC1 , t _C1 ) from the No. 1 side view camera coordinate system O _C1 to the global coordinate system O _G. R _C1 is the No. 1 side view camera coordinate system O _C1 to the global coordinate system O G. The rotation matrix of the coordinate system O _G , t _C1 is the translation vector from the No. 1 side view camera coordinate system O _C1 to the global coordinate system O _G ;

And the conversion relationship between the No. 2 side-view camera coordinate system O _C2 and the global coordinate system O _G is T _C2 = ( _RC2 ,t _C2 ). R _C2 is the conversion relationship between the No. 2 side-view camera coordinate system O _C2 and the global coordinate system O _G. Rotation matrix, t _C2 is the translation vector from the No. 2 side view camera coordinate system O _C2 to the global coordinate system O _G.

See Figure 2 for the external parameter matrix calibration. In the external parameter calibration, since the camera coordinate system of the global camera is defined as a unified global coordinate system, only the external parameters of the two side-view cameras need to be calibrated, which is an orthogonal visual inspection system. For the calibration of external parameters, a three-dimensional calibration block is designed specifically and combined with the AprilTag algorithm calibration. The designed calibration block is 3D printed with resin material and has a rectangular shape. Its size specifications are shown in Figure 2, and the processing accuracy is ±0.05mm. The three-dimensional calibration block is a cuboid. The four sides and the top surface are defined as surface 0-surface 4 of the cube in sequence. Surface 0-surface 4 adopt the 36H11 AprilTag pattern with id 0-4 in sequence. The coordinate system of the three-dimensional calibration block is O _B The origin is defined at the center of the AprilTag pattern (id=0) on surface 0. The X-axis and Y-axis are parallel to the surface. The positive directions are the direction from the upper left vertex of the pattern to the upper right vertex, the direction from the upper left vertex of the pattern to the lower left vertex, and the Z axis direction. Inside the model;

The process of calibrating the external parameter matrix using the AprilTag algorithm is:

When calibrating the external parameters of the side-view camera, the arrangement and placement of the cuboid calibration blocks meet the working distance requirements of the camera when detecting optical components. Keep the surface 4 of the three-dimensional calibration block horizontal. Select one side from the surfaces 0-3 of the three-dimensional calibration block. Let the distance between the AprilTag pattern on the selected surface and the optical center of the No. 1 side-view camera along the optical axis direction be 280mm. The AprilTag pattern on the surface 4 The distance to the optical center of the global camera along the optical axis is 1200mm, and the corresponding AprilTag pattern must appear completely in the fields of view of the above two cameras;

When the stereo calibration block, the global camera, and the No. 1 side view camera meet the above requirements, use the external trigger method to control the two cameras to capture images with a time difference of no more than 15ms, and use the AprilTag algorithm to calculate the positions of the calibration graphics in the field of view to the respective cameras. Attitude; the transformation relationship between the coordinate system of all graphics on the stereo calibration block and the coordinate system O _B of the calibration block is known from the CAD data. The pose of the calibration block relative to the global camera and the No. 1 side view camera at the time of taking the photo is deduced. Solution Calculate the transformation between these two poses T _C1 = (R _C1 ,t _C1 ), which is the conversion relationship from O _C1 to O _G , and obtain the external parameter matrix of side view camera No. 1 under O _G ;

The extrinsic parameter matrix of No. 2 side-view camera under O _G is obtained in the same way as the above process. The conversion relationship from the No. 2 side view camera coordinate system _OC2 to the global coordinate system _OG is _TC2 = ( _RC2 , t _C2 ).

Regarding step S3, the optical element pose solution process is:

S31. Use the fixture to place the optical element at the predetermined position on the FOA platform, let the three cameras simultaneously collect the top view image I _M and the two side view images I _SL and I _SF of the optical element, and preprocess the three images to generate a grayscale image. ;

Specifically, first, the manipulator clamps the optical element to a predetermined position, turns on the grazing light source, adjusts the lighting angle so that the edge of the optical element is illuminated in the camera's field of view, and adjusts it according to the average pixel gray value of the preset RoI area. The camera exposure time is such that the corresponding pixel grayscale (value range is 0-255) at the edge of the optical element is between 220-240; the three cameras are controlled through external triggering to simultaneously collect images within 150ms, and the collected images are global cameras. The image I _M has a resolution of 5120×5120; the side view camera images I _SL and I _SF have a resolution of 4200×2160.

S32. Use the LSD algorithm to extract line segments from the three grayscale images, process these line segments, and obtain the edges of the optical elements of the three grayscale images in the respective camera pixel coordinate systems;

The acquisition process of the edge of the optical element is:

The LSD algorithm is used to extract all line segments from the three grayscale images and processed as follows:

Step 1. Eliminate too short line segments with L<25 pixels. The length of the line segment L is obtained as follows:

(x ₁ , y ₁ ), (x ₂ , y ₂ ) are the pixel coordinates of the two end points of the line segment respectively;

Step 2: Record the set of unfused line segments as l _Ori and the set of fused line segments as l _meg . The remaining line segments after eliminating the short line segments in step 1 are first placed in the set l _Ori ;

Step 3. Fusion of line segments:

Search for the line segment with the largest L in the set l _Ori , recorded as l ₀ , and search for the line segment that meets the conditions in the set l _Ori : the Euclidean distance dis between the midpoint of the line segment and the straight line where l ₀ is less than 3 pixels, and the angle between the line segment and l ₀ ang is less than 0.5°; remove l ₀ and qualified line segments from the set l _Ori ;

The Euclidean distance dis and angle ang between two line segments are calculated as follows:

in:

(a ₁ ,b ₁ ), (a ₂ ,b ₂ ) are the endpoint coordinates of l ₀ ,

k is the slope of l ₀ ,

x＝(x ₁ +x ₂ )/2

y＝(y ₁ +y ₂ )/2

Δx＝x ₂ -x ₁

Δy＝y ₂ -y ₁

Δa＝a ₂ -a ₁

Δb＝b ₂ -b ₁

Step 4. For l ₀ and qualified line segments, use the SMBR algorithm to calculate the minimum circumscribed rectangle of the endpoints of these line segments, and use the pixel coordinates of the midpoint of the two short sides of the circumscribed rectangle as the pixel coordinates of the two endpoints of the new fused line segment. Put this fused line segment into the fused line segment set l _meg ;

Repeat steps three and four until the set l _Ori is an empty set, and then perform step five;

Step 5. Delete the wrong line segments in the set l _meg :

First, delete the line segments with L < 1200 pixels; since the length of the line segments at the edge of the optical components in the real-shot images is generally longer than 1200 pixels, the line segments with L < 1200 pixels are deleted, that is, the line segments that are too short are deleted.

Secondly, the prior knowledge of the optical element CAD model is used to identify the line segments belonging to the edge of the optical element. Specifically, the optical element clamped by the fixture is reset to a predetermined position, and the CAD of the optical element at this time is projected into the collected image. Since the extracted line segments come from the edge of the optical element, the line segment that correctly represents the edge needs to meet the following two conditions at the same time (that is, the line segments representing the edge of the optical element are filtered according to the following conditions): The midpoint of the line segment is consistent with a certain model extracted from CAD The Euclidean distance dis of the edge line segment is less than 15 pixels, and the included angle ang is less than 2°; the line segments in the set l _meg that do not meet the above conditions are deleted, and the remaining line segments in the set l _meg are used as the edge of the optical element.

S33. Obtain the analytical formula of the edge of the optical element in the three camera coordinate systems. Specifically, according to the matching result of step S32, when an edge in the top-view image _IM matches an edge in the side-view image I _SL or the side-view image I _SF If a certain edge simultaneously corresponds to the same edge in the CAD model, it is considered that the edges in the above two images correspond to the same edge of the optical element; according to this principle, the analytical formulas of the three grayscale images in their respective camera coordinate systems are obtained;

S34. According to the camera internal parameter matrix, external parameter matrix, and the analytical formulas in the three camera coordinate systems, obtain the analytical formula of the edge of the optical element in the global coordinate system;

The process of obtaining the analytical expression of the edge of an optical element in the global coordinate system is:

Referring to Figure 3, firstly, the principle description of edge projection in the pixel coordinate system in three-dimensional space is given. According to the known internal parameter matrix and external parameter matrix of the camera, the straight line corresponding to the same edge of the optical element in the two images is back-projected. Convert to the plane under the camera coordinate system to obtain the three-dimensional parameters of the plane where the straight line is located, which passes through the camera optical center O; the two images are a top view and a side view; the two images correspond to the optical element to be tested The straight line at the same edge is converted to a plane under the camera coordinate system using back projection, and the three-dimensional parameters of the plane where the straight line is located are obtained, and the plane passes through the camera optical center O. As shown in Figure 3, x ₁ and x ₂ are two points on a straight line in the image, which are known quantities; X ₁ and X ₂ are the corresponding points of these two points in the three-dimensional space, and the space of X ₁ and X ₂ The coordinates are unknown, and cannot be known even if the calibration data of the camera is used. However, according to the internal parameters of the camera, it can be determined through projection that the spatial points X ₁ and X ₂ are on the plane passing through the points O, x ₁ and x ₂ , that is, the camera pixel A line segment (straight line) on the plane corresponds to a surface in three-dimensional space.

Assume that the edge line segment l ₁ from the global camera and the edge line segment l ₂ of a side view camera correspond to the same straight line in space, l ₁ and the optical center of the global camera determine the plane P ₁ , l ₂ and the optical center of the side view camera determine For plane P ₂ , planes P ₁ and P ₂ are transformed into the global coordinate system according to the internal parameter matrix and external parameter matrix, and the analytical formulas of P ₁ and P ₂ in the global coordinate system are obtained:

P ₁ :a ₁ X+b ₁ Y+c ₁ Z+d ₁ =0

P ₂ :a ₂ X+b ₂ Y+c ₂ Z+d ₂ =0

Among them, the vector (a ₁ , b ₁ , c ₁ ) is the normal vector of the plane P ₁ , d ₁ is the suffix that satisfies all points on the plane P ₁ ; the vector (a ₂ , b ₂ , c ₂ ) is the normal vector of the plane P ₂ Vector, d ₂ is the suffix that satisfies all points on the plane P ₂ ; c ₁ =c ₂ =1;

Since in the imaging of two cameras, the matching straight line corresponds to the same straight line in the three-dimensional space, the planes P ₁ and P ₂ must intersect, and the intersection line is the target straight line, that is, the analytical formula of the straight line where the edge of the optical element is located is:

P＝p ₀ +tV

Where p ₀ = (X ₀ , Y ₀ , Z ₀ ) is a known point on the target straight line, t is the unit direction vector of the target straight line, in the form of t=iX+jY+kZ, coefficients i, j, k Satisfy the relationship: i ² +j ² +k ² =1;

According to the above principle, the analytical formula of the straight line in the global coordinate system where the upper surface of the optical element is close to the two adjacent edges of the two side-view cameras can be obtained:

L ₁ :P＝p ₁ + ₁ V

L ₂ :P＝p ₂ + ₂ V

Among them, p ₁ is a known point on the straight line L ₁ , t ₁ is the unit direction vector of the straight line L ₁ , p ₂ is a known point on the straight line L ₂ , t ₂ is the unit direction vector of the straight line L ₂ ;

Calculate the point _A = (X ^M , Y ^M _, Z ^M ) where the sum of the distances to L ₁ and L ₂ is the smallest Points B and C are 410mm apart, and a plane is determined by A, B, and C:

P ₀ :a ₀ X+b ₀ Y+Z+c ₀ =0

That is the analytical formula of the upper surface of the optical element in the global coordinate system. According to the above method, the intersection line between the upper surface and the plane where the other two edges are located is solved, which is the spatial analytical formula of the straight line where the remaining two edges are located in the global coordinate system. :

L ₃ : P＝p ₃ + ₃ V

L ₄ : P＝p ₄ + ₄ V

Among them, p ₃ is a known point on the straight line L ₃ , t ₃ is the unit direction vector of the straight line L ₃ , p ₄ is a known point on the straight line L ₄ , and t ₄ is the unit direction vector of the straight line L ₄ .

S35. According to the analytical formula of the edge of the optical element in the global coordinate system, use the PnP algorithm (Perspective-n-Point, an algorithm for solving the motion of 3D to 2D point pairs) to calculate the position, posture and size of the upper surface of the optical element according to the corresponding relationship. .

The process of solving the position and attitude of the optical element and the size of the upper surface is:

The positive Z-axis direction of the attitude from the optical element model coordinate system O _M to the global coordinate system O _G is defined as t ₂ ×t ₁ , and the positive X-axis direction is defined as t ₂ . Construct PnP correspondence:

The two-dimensional point (0,0) corresponds to the three-dimensional point A=(X _M , Y _M , Z _M );

The two-dimensional point (1,0) corresponds to the three-dimensional point A+t ₂ ;

Two-dimensional point (0,1) corresponds to three-dimensional point A+Norm(t ₂ ×t ₁ )×t ₂ ;

Two-dimensional point (1,1) corresponds to three-dimensional point A+Norm(t ₂ ×t ₁ )×t ₂ +t ₂ ;

Among them, Norm(t) is the operation of normalizing t;

By establishing the PnP relationship of the above four groups of points, the conversion relationship (R, t) from the model coordinate system O _M of the optical element to the global coordinate system O _G can be solved. (R, t) is used as the 6-degree-of-freedom position of the optical element. pose, R is the rotation matrix from the model coordinate system O _M of the optical element to the global coordinate system O _G , t is the translation vector from the model coordinate system O _M of the optical element to the global coordinate system O _G ;

Position matrix t=A ^T =(X _M , Y _M , Z _M ) ^T ;

According to the method of solving the coordinates of point A in the global coordinate system, obtain the coordinates of the remaining vertices on the upper surface of the optical element in the global coordinate system. The remaining vertices are B, C, and D in clockwise order.

The length of the optical element is L=||((A+B)-(D+C))/2|| ₂ and the width is W=||((A+D)-(B+C))/2| | ₂ .

Specific Embodiment 2: This embodiment will be described below with reference to Figure 1. The optical element assembly method based on the orthogonal vision system described in this embodiment is implemented based on the detection method of Embodiment 1. The optical element six calculated in Embodiment 1 is free. The pose is applied to the subsequent assembly process. During assembly, the pose detection method of the optical component assembly process based on the orthogonal vision system described in Embodiment 1 is used to obtain the pose of the optical component in real time, combined with the hand-eye calibration information of the manipulator and The ideal posture that the optical element should reach is calculated, and the deviation between the current optical element's posture and the ideal posture is calculated, and this deviation is used to guide the manipulator to adjust its posture.

Although the present invention is described herein with reference to specific embodiments, it is to be understood that these embodiments are merely exemplary of the principles and applications of the invention. It is therefore to be understood that many modifications may be made to the exemplary embodiments and other arrangements may be devised without departing from the spirit and scope of the invention as defined by the appended claims. It is to be understood that the features described in the different dependent claims may be combined in a different manner than that described in the original claims. It will also be understood that features described in connection with individual embodiments can be used in other described embodiments.

Claims (9)

1. The method for detecting the pose of the optical element assembly process based on the orthogonal vision system is characterized by comprising the following steps of:

s1, constructing a visual detection system;

the visual detection system comprises a global camera and two side view angle cameras, wherein the front of the two side view angle cameras uses a strip light source to glancing and illuminating the optical element;

s2, a joint calibration step of three cameras in the visual detection system;

taking a camera coordinate system of a global camera as a unified global coordinate system; three camera internal parameters and external parameters are marked, and a spatial mapping relation from a pixel coordinate system of each camera to a unified global coordinate system is established;

s3, calculating the pose of the optical element;

the three cameras synchronously acquire a top view and two side views of the optical element, extract the analysis of the edges of the optical element under the respective pixel coordinate systems from the images of the three cameras, and align the edges under the pixel coordinate systems of the different cameras under a unified global coordinate system by combining the combined calibration data; the pose of the optical element is determined by the analytical formula of these edges.

2. The method for detecting the pose of an optical element assembly process based on an orthogonal vision system according to claim 1, wherein the optical element to be detected is horizontally arranged on a FOA assembly platform, a global camera is arranged above the FOA assembly platform, two cameras positioned at the side of the optical element during detection are a No. 1 side view angle camera and a No. 2 side view angle camera,

the three cameras are orthogonal to each other in terms of gestures, the optical axes of the three cameras intersect at one point on the upper surface of the optical element, and the following coordinate system is established:

the global coordinate system of the global camera is O _G -X _G Y _G Z _G The optical axis direction of the global camera is vertical to the horizontal plane;

no. 1 side view camera coordinate system O _C1 -X _C1 Y _C1 Z _C1 No. 2 side view camera coordinate system O _C2 -X _C2 Y _C2 Z _C2 The optical axis directions of the two cameras are parallel to the horizontal plane;

global camera coordinate system O _G X-axis forward direction and side view angle camera coordinate system O number 2 _C2 Z of (2) _C2 The positive directions of the axes are the same;

model coordinate system O of optical element _M Is O _M -X _M Y _M Z _M And the origin is defined at the vertex A of the upper left corner of the upper surface of the optical element, Z _M The axis is perpendicular to the upper surface of the optical element, Z _M Upward in the positive direction of axis, X _M The positive axis direction is the direction in which the vertex of the upper left corner of the upper surface of the optical element points to the vertex of the lower left corner of the upper surface of the optical element.

3. The method for detecting the pose of an optical element assembly process based on an orthogonal vision system according to claim 2, wherein the joint calibration process of step S2 is as follows:

firstly, calibrating internal reference matrixes of three cameras respectively by adopting a Zhang Zhengyou calibration method, and establishing a mapping relation from a two-dimensional pixel coordinate system to a three-dimensional camera coordinate system;

then, the camera coordinate systems of the global cameras are used as a unified global coordinate system, and the transformation from the camera coordinate systems of the two side view angle cameras to the coordinate systems of the global coordinate system is respectively calibrated to obtain external parameter matrixes of the two side view angle cameras;

the extrinsic matrix comprises: no. 1 side view angle camera coordinate system O _C1 To the global coordinate system O _G Is a conversion relation T of (2) _C1 ＝(R _C1 ,t _C1 )，R _C1 No. 1 side view camera coordinate system O _C1 To the global coordinate system O _G T _C1 No. 1 side view camera coordinate system O _C1 To the global coordinate system O _G Is a translation vector of (a);

no. 2 side view angle camera coordinate system O _C2 To the global coordinate system O _G Is T _C2 ＝(R _C2 ,t _C2 )，R _C2 No. 2 side view camera coordinate system O _C2 To the global coordinate system O _G T _C2 No. 2 side view camera coordinate system O _C2 To the global coordinate system O _G Is a translation vector of (a).

4. The method for detecting the pose of an optical element assembly process based on an orthogonal vision system according to claim 2 or 3, wherein a three-dimensional calibration block is adopted and combined with an april tag algorithm to calibrate the external reference matrixes of two side-view angle cameras, the three-dimensional calibration block is a cuboid, four side surfaces and the top surface are sequentially defined as surfaces 0-4 of the cuboid, the surfaces 0-4 sequentially adopt 36H11 april tag patterns with id of 0-4, and the coordinate system O of the three-dimensional calibration block _B The origin defines the center of an april tag pattern (id=0) on the surface 0, the X axis and the Y axis are parallel to the surface, the positive directions are the directions from the top left vertex to the top right vertex of the pattern, from the top left vertex to the bottom left vertex of the pattern, and the Z axis direction points to the inner side of the model;

the process of calibrating the external reference matrix by adopting the april tag algorithm comprises the following steps:

the surface 4 of the three-dimensional calibration block is kept horizontal, one surface is arbitrarily selected from the surfaces 0-3 of the three-dimensional calibration block, the distance from the april tag pattern of the selected surface to the optical center of the No. 1 side view angle camera along the optical axis direction is 280mm, the distance from the april tag pattern of the surface 4 to the optical center of the global camera along the optical axis direction is 1200mm, and the corresponding april tag pattern is ensured to completely appear in the visual fields of the two cameras;

when the three-dimensional calibration block, the global camera and the No. 1 side view angle camera meet the requirements, an external triggering mode is used for controlling the two cameras to capture images under the time difference of not more than 15ms, and an april tag algorithm is used for calculating the position and the gesture from the calibration graph in the view field to each camera; coordinate system of all graphics on three-dimensional calibration block and coordinate system O of calibration block _B The transformation relation of the camera is known by CAD data, the pose of the shooting moment calibration block relative to the global camera and the No. 1 side view angle camera is deduced, and the transformation T between the two poses is calculated _C1 ＝(R _C1 ,t _C1 ) Namely O _C1 To O _G Acquiring the conversion relation of the No. 1 side view angle camera at O _G A lower extrinsic matrix;

no. 2 side view angle camera at O _G The acquisition mode of the external reference matrix is the same as the above process.

5. The method for detecting the pose of an optical element assembly process based on an orthogonal vision system according to claim 3, wherein the pose calculation process of the optical element in the step S3 is as follows:

s31, placing the optical element at a preset position of the FOA assembly platform by using a clamp, so that three cameras synchronously acquire overlook images I of the optical element _M And two side view images I _SL 、I _SF Preprocessing the three images to generate a gray level image;

s32, extracting line segments from the three gray images by adopting an LSD algorithm, and processing the line segments to obtain the edges of the optical elements of the three gray images under the respective camera pixel coordinate systems;

s33, acquiring the analytic expressions of the edges of the optical element under three camera coordinate systems, specifically, according to the matching result in the step S32, when looking down the image I _M One edge of the image I _SL Or side view image I _SF One edge of the CAD model corresponds to the same edge at the same time, and then the CAD model considers thatEdges in the two images correspond to the same edge of the optical element; according to the principle, obtaining analytic formulas of three gray images under respective camera coordinate systems;

s34, according to the camera internal reference matrix, the external reference matrix and the analysis formula under the three camera coordinate systems, the analysis formula of the edge of the optical element under the global coordinate system is obtained;

s35, according to the analysis of the edge of the optical element under the global coordinate system, calculating the position and the posture of the optical element and the size of the upper surface according to the corresponding relation by utilizing a PnP algorithm.

6. The method for detecting the pose of an optical element assembly process based on an orthogonal vision system according to claim 5, wherein the process of obtaining the edge of the optical element in step S32 is as follows:

all line segments are extracted from the three gray images by adopting an LSD algorithm, and the following processing is carried out:

step one, eliminating the excessively short line segment of L <25 pixels, wherein the length L of the line segment is obtained according to the following formula:

(x ₁ ,y ₁ ),(x ₂ ,y ₂ ) Pixel coordinates of two endpoints of the line segment respectively;

step two, marking the segment set which is not fused as l _Ori The fused set of line segments is denoted as l _meg The rest line segments after the short line segments are removed in the first step are firstly placed in the set l _Ori In (a) and (b);

step three, fusing line segments:

in set l _Ori The line segment with the largest L is searched and marked as L ₀ In set l _Ori A line segment meeting the condition: line segment midpoint to l ₀ The linear Euclidean distance dis is smaller than 3 pixels, and the line segment is equal to l ₀ The included angle ang is smaller than 0.5 degrees; will l ₀ Line segment slave set meeting condition _Ori Removing from the middle part;

the Euclidean distance dis and the included angle ang of the two line segments are calculated according to the following formula:

wherein:

(a ₁ ,b ₁ ),(a ₂ ,b ₂ ) Is l ₀ The coordinates of the end points,

k is l ₀ The slope of the slope is calculated,

x＝(x ₁ +x ₂ )/2

y＝(y ₁ +y ₂ )/2

Δx＝x ₂ -x ₁

Δy＝y ₂ -y ₁

Δa＝a ₂ -a ₁

Δb＝b ₂ -b ₁

step four, for l ₀ And the minimum circumscribed rectangle of the line end points is calculated by using the SMBR algorithm, the midpoint pixel coordinates of the two short sides of the circumscribed rectangle are used as the pixel coordinates of the two end points of the new fused line, and the fused line is put into the fused line set l _meg In (a) and (b);

repeating the third and fourth steps until the set l _Ori The set is empty, and then the fifth step is executed;

step five, deleting the set l _meg Middle error line segment:

first, deleting the line segment of L <1200 pixels;

second, using a priori knowledge of the CAD model of the optical element to identify line segments belonging to the edge of the optical element, in particular, the optical element to be clamped by the clampResetting to a preset position, and projecting CAD of the optical element at the moment into the acquired image. Since the extracted line segments are from the optical element edges, the line segments representing the optical element edges are screened as follows: the Euclidean distance dis from the midpoint of the line segment to the edge line segment of a certain model extracted by CAD is smaller than 15 pixels, and the included angle ang is smaller than 2 degrees; will aggregate l _meg Segment deletion not meeting the above conditions, set l _meg The remaining line segments of (a) act as optical element edges.

7. The method for detecting pose of an optical element assembling process based on an orthogonal vision system according to claim 6, wherein the step S34 of obtaining the resolution of the edge of the optical element in the global coordinate system is:

according to the known camera internal reference matrix and external reference matrix, converting the straight line of the same edge of the corresponding optical element in the two images into a plane under a camera coordinate system by using back projection to obtain the three-dimensional parameter of the plane where the straight line is located, wherein the plane passes through a camera optical center O; the two images are a top view and a side view;

assume an edge line segment l from a global camera ₁ And an edge line segment l of a side view camera ₂ Corresponding to the same straight line in the space, l ₁ And the optical center determination plane P of the global camera ₁ ，l ₂ And a light center determination plane P of a side view angle camera ₂ The plane P is divided into an internal reference matrix and an external reference matrix ₁ 、P ₂ Transforming to global coordinate system to obtain P ₁ 、P ₂ The analytical formula under the global coordinate system is:

P ₁ :a ₁ X+b ₁ Y+c ₁ Z+d ₁ ＝0

P ₂ :a ₂ X+b ₂ Y+c ₂ Z+d ₂ ＝0

wherein vector (a) ₁ ,b ₁ ,c ₁ ) Is a plane P ₁ Normal vector d of (d) ₁ Is to satisfy all points in plane P ₁ Adding a suffix; vector (a) ₂ ,b ₂ ,c ₂ ) Is a plane P ₂ Normal vector d of (d) ₂ Is to satisfy allPoint at plane P ₂ Adding a suffix; c ₁ ＝c ₂ ＝1；

Because in the imaging of the two cameras, the matched straight lines correspond to the same straight line in the three-dimensional space, the plane P ₁ And P ₂ The intersecting lines are the target straight lines, namely the straight line analysis of the edges of the optical element is as follows:

P＝p ₀ +tV

wherein p is ₀ ＝(X ₀ ，Y ₀ ，Z ₀ ) Is a known point on the target line, t is the unit direction vector of the target line, the form is t=ix+ jY +kz, and the coefficients i, j, k satisfy the relation: i.e ² +j ² +k ² ＝1；

By repeating the method, the analytical formula of the straight line where the upper surface of the optical element is close to the adjacent two edges of the two side view angle cameras in the global coordinate system can be obtained:

L ₁ ：P＝p ₁ +t ₁ V

L ₂ ：P＝β ₂ +t ₂ V

wherein p is ₁ Is a straight line L ₁ A known point, t ₁ Is a straight line L ₁ Unit direction vector, p ₂ Is a straight line L ₂ A known point, t ₂ Is a straight line L ₂ Is a unit direction vector of (a);

solving to L ₁ And L ₂ Point a= (X) where the sum of distances is minimum ^M ，Y ^M ，Z ^M ) Is the top left corner apex of the upper surface of the optical element, i.e. at L ₁ And L ₂ Taking a point B and a point C with a distance of 410mm from the point A respectively, and jointly determining a plane by A, B, C:

P ₀ ：a ₀ X+b ₀ Y+Z+c ₀ ＝0

the method is that the analytic expression of the upper surface of the optical element under the global coordinate system is that the intersection line of the upper surface and the planes of the other two edges is solved according to the method, namely the spatial analytic expression of the straight line of the remaining two edges under the global coordinate system is that:

L ₃ ：P＝p ₃ +t ₃ V

L ₄ ：P＝p ₄ +t ₄ V

wherein p is ₃ Is a straight line L ₃ A known point, t ₃ Is a straight line L ₃ Unit direction vector, p ₄ Is a straight line L ₄ A known point, t ₄ Is a straight line L ₄ Is a unit direction vector of (a).

8. The method for detecting the pose of an optical element assembly process based on an orthogonal vision system according to claim 7, wherein the step of S35 of calculating the position pose of the optical element and the size of the upper surface is:

optical element model coordinate system O _M To the global coordinate system O _G Is defined as t ₂ ×t ₁ The positive X-axis direction is defined as t ₂ . Constructing PnP corresponding relation:

two-dimensional point (0, 0), corresponding to three-dimensional point a= (X) _M ，Y _M ，Z _M )；

Two-dimensional point (1, 0) corresponding to three-dimensional point A+t ₂ ：

Two-dimensional points (0, 1) corresponding to three-dimensional points A+Norm (t ₂ ×t ₁ )×t ₂ ；

Two-dimensional points (1, 1) corresponding to three-dimensional points A+Norm (t ₂ ×t ₁ )×t ₂ +t ₂ ；

Wherein Norm (t) is the operation normalized to t;

by establishing PnP corresponding relation of the four groups of points, a model coordinate system O of the optical element can be solved _M To the global coordinate system O _G The conversion relation (R, t), (R, t) of the optical element is taken as the 6-degree-of-freedom pose of the optical element, and R is taken as the model coordinate system O of the optical element _M To the global coordinate system O _G T is the model coordinate system O of the optical element _M To the global coordinate system O _G Is a translation vector of (a);

position matrix t=a ^T ＝(X _M ,Y _M ,Z _M ) ^T ；

Acquiring the coordinates of the residual vertex of the upper surface of the optical element under the global coordinate system according to the solving method of the coordinates of the point A under the global coordinate system, wherein the residual vertex is B, C, D in turn clockwise,

the length of the optical element is l= | ((a+b) - (d+c))/2|| ₂ The width is w= | ((a+d) - (b+c))/2|| ₂ 。

9. The method for assembling the optical element based on the orthogonal vision system is characterized in that the pose of the optical element is obtained in real time by using the pose detection method for the optical element assembling process based on the orthogonal vision system according to claims 1-8 during assembly, the deviation between the pose of the current optical element and the ideal pose is calculated, and the manipulator is guided to adjust the pose.