CN116625258A

CN116625258A - Chain spacing measuring system and chain spacing measuring method

Info

Publication number: CN116625258A
Application number: CN202310549713.2A
Authority: CN
Inventors: 徐锟; 王舒扬
Original assignee: MCC Baosteel Technology Services Co Ltd
Current assignee: MCC Baosteel Technology Services Co Ltd
Priority date: 2023-05-16
Filing date: 2023-05-16
Publication date: 2023-08-22

Abstract

The chain distance measuring system and the chain distance measuring method are characterized in that at least two sets of machine vision sensor systems are erected on two sides of the chain distance to be measured, each machine vision sensor system comprises a binocular vision system consisting of two cameras, and each four-vision system comprises a calibration unit and a measuring unit; the calibration unit comprises a three-dimensional coding target and is used for calibrating a four-eye vision system; the calibration unit comprises a high-precision target rod and is used for detecting the system precision of the four-vision system; when the chain runs, the four-eye vision system acquires chain images at intervals, automatically measures the center of a chain shaft in the images through elliptical detection based on deep learning, and the measuring unit is used for continuously and automatically calculating the distance between the chain shafts and the offset angle of the chain shafts; the chain distance measuring method comprises a calibration step and a measuring step.

Description

Chain spacing measuring system and chain spacing measuring method

Technical Field

The invention relates to the technical field of visual image detection application of transmission chain spacing, in particular to a chain spacing measuring system.

Background

The activated carbon conveying system is a special equipment system for conveying activated carbon and mainly comprises a storage container, a conveying pipeline, a conveying machine and the like; the existing activated carbon conveying system drives the whole equipment to operate through a chain; however, in the actual use process, the intervals between the chains are gradually increased due to long-time abrasion; when the distance between the chains is increased to a certain distance, the whole chain is broken, and then the whole activated carbon conveying system stops running and waits for a new chain to be replaced; the process can very influence the production efficiency and the economic benefit;

The transmission chain interval is the distance between two adjacent pin shaft centers of the transmission chain, the size of the interval has great influence on the operation and the transmission efficiency of the transmission chain, the operation and the transmission efficiency are required to be monitored, but the manual operation and the detection are mainly relied on at present, the time and the labor are wasted, the accuracy is poor, and the on-line monitoring cannot be realized.

In the prior art 1, an artificial intelligence-based automobile model visual detection method is described in Chinese patent application number CN202211380160.4, which is characterized in that a visual detection module is arranged to collect left paint image data, right paint image data, front paint image data, rear paint image data and paint image data of an automobile model, an image processing module is arranged to define abnormal pixel sections based on each row in the paint image data, an abnormal pixel area of the automobile model is screened and defined, a defect alarm module is arranged to alarm and display a defect area to a detector based on the abnormal pixel area, the damage degree of the paint of the automobile model is thinned, the accuracy of the paint defect detection of the automobile model is increased, and the detection efficiency is accelerated;

in the prior art 2, a three-dimensional sparse reconstruction method, a three-dimensional sparse reconstruction device and an electronic device of Chinese patent application number CN202211185769.6 are described, wherein urban aerial images acquired by a multi-camera are acquired, the aerial images are input into a global feature extraction network, and feature values are extracted; the first determining module is used for processing the characteristic values by adopting a first clustering algorithm and determining a first image partition; the acquisition module is used for acquiring a feature matching result by carrying out feature matching on the image pairs in the first image partition, wherein the image in the first image partition can be determined according to the feature matching result; the second determining module is used for reconstructing the image in the first image partition, determining a first camera pose, and carrying out iterative processing on the first camera pose by adopting a second clustering algorithm based on the first image partition to determine the camera pose;

In the prior art 1, a multi-camera is adopted to collect left paint image data, right paint image data, front paint image data, rear paint image data and paint image data of an automobile model, an image processing module processes full paint image data of the automobile model according to a certain processing mode to generate full paint defect data of the automobile model, and a defect alarm module carries out defect alarm and defect area display on detection personnel according to the full paint defect data of the automobile model.

In the prior art 2, urban aerial images acquired by a multi-view camera are input into a global feature extraction network, and the feature values are divided and spliced by using a clustering algorithm.

However, the prior art cannot solve and realize real-time online monitoring of the distance between two adjacent pin centers of the transmission chain.

Disclosure of Invention

In view of the above, the present invention aims to provide a portable chain distance measuring system for monitoring and measuring structural shape parameters of a chain in real time, generating a parameter report of an integral chain, and feeding back detected abnormal data of the chain to a technical manager, so that the manager can conveniently know the running condition of the whole set of chain, namely the chain distance measuring system.

The present application aims to solve one of the problems in the background art.

The technical scheme adopted by the application is as follows: in order to achieve the above and other related objects, the present application provides a chain pitch measurement system, wherein at least two sets of machine vision sensor systems are erected on two sides of a chain pitch to be measured, the machine vision sensor systems comprise a binocular vision system composed of two cameras, and a four-vision system comprises a calibration unit and a measurement unit;

the calibration unit comprises a three-dimensional coding target and is used for calibrating a four-eye vision system;

the calibration unit comprises a high-precision target rod and is used for detecting the system precision of the four-vision system;

when the chain runs, the four-vision system acquires chain images at intervals, and automatically measures the center of a chain shaft in the images through elliptical detection based on deep learning, and the measuring unit is used for continuously and automatically calculating the distance between the chain shafts and the offset angle of the chain shafts.

Preferably, the four-vision system recognizes, tracks and locates the chain shaft; when the tested chain is just in the central region of the image, the center of the chain shaft is extracted, and the elliptic regression network of the mask-CNN is directly fitted to the outline of the ellipse.

Preferably, a high-precision target rod is arranged in a matched manner and used for automatically detecting the system precision of the four-vision system.

Preferably, the calibration unit is run on the device every 2 or 3 months.

Preferably, the four cameras of the two binocular systems constitute a four-vision system.

Preferably, a laser and a light source are arranged in a matched manner, so that the calculation of the three-dimensional coordinates of the center of the chain shaft is realized.

Preferably, a blowing device is provided and an industrial brush module is installed for blowing and cleaning the chain shaft.

Preferably, a tripod is provided for supporting the stationary camera and the light source.

Preferably, the tripod is height-adjustable.

The chain distance measuring method, which is applied to the chain distance measuring system, comprises the following steps: a calibration step and a measurement step;

the calibration steps are as follows: an operator holds the corresponding three-dimensional coding target within 0.3-1 meter from a camera of a four-vision system, more than 15 images are collected, the four-vision system guides the operator how to collect the images, and the four-vision system automatically completes the calibration of the four-vision system under a non-overlapping vision field according to the collected target images;

the measuring steps are as follows: when the chain runs, the four-vision system acquires chain images at intervals, and the distance between the chain shafts and the offset angle of the chain shafts are continuously and automatically calculated by automatically measuring the center of the chain shafts in the images through elliptical detection based on deep learning.

Preferably, a set of four-vision system is used for accurately monitoring the distance between the driving chains in real time.

The chain distance measuring system is arranged at a position along the line which does not influence the movement of the chain and comprises a plurality of devices which are discontinuously arranged;

two sets of machine vision sensor systems are erected on two sides of the chain spacing to be measured, and each machine vision sensor system comprises a group of binocular vision systems consisting of two industrial cameras;

the four cameras together form a four-eye vision system under a non-overlapping visual field, and a laser and a light source are arranged in a matched manner to provide additional three-dimensional information or peripheral peripherals of a good image acquisition environment for realizing calculation of a three-dimensional coordinate of the center of a chain shaft;

the two cameras are placed on the special tripod and fixed by the tripod below, and the whole height of the tripod can be freely telescopic and adjustable to form a mobile portable binocular system;

two sides of the chain shaft are respectively provided with a binocular system, and four cameras of the two binocular systems form a four-vision system;

a light source is additionally arranged near the annular area outside the camera, and the on-site working condition is complex and dim and is used for overcoming the influence of the working environment on the imaging quality;

the blowing equipment is arranged, and the industrial hairbrush module is arranged and used for blowing and cleaning the chain shaft, so that the influence of dust in the working environment on the chain image is eliminated, and the imaging quality of a camera is ensured;

The operation of the four-vision system comprises a calibration unit and a measurement unit;

calibration unit stage:

calibration in vision measurement means that the camera and the image processing system are calibrated and configured so that the system can accurately measure the size and position of an object; before a measurement is made, calibration must be performed to obtain an accurate measurement result; calibration typically involves determining camera internal and external parameters, including focal length, optical center, distortion coefficients of the camera, and geometric relationships between the camera and the object; through calibration, the system can correct the image and convert the pixel value in the image into physical quantity in the real world, thereby realizing high-precision measurement;

the calibration of the four-vision system comprises the following specific steps: after the equipment is erected, an operator holds the corresponding three-dimensional coding target and acquires 15-20 images according to the requirement at a position about 0.6 meter away from the equipment camera, and in the process, a four-vision system automatically guides the operator how to acquire the images, so that the four-vision system automatically completes the calibration of the four-vision system under a non-overlapping visual field according to the acquired target images;

the non-overlapping view field refers to a part where two adjacent images are not overlapped when photographing or shooting; meaning that the scene captured by two adjacent images is different, with no duplicate portions; the non-overlapping view fields can obtain a complete, continuous and seamless spliced image by shooting images at different positions for multiple times and then splicing the images together; a wider field of view, higher resolution and more detailed details can be achieved;

After the calibration process is skilled, the time for drawing and calibrating the whole system is less than 5 minutes, and the process can be completed by a single person only;

after the calibration is finished, the accuracy of the equipped high-accuracy target rod automatic detection system can be used; the calibration process is not required to be carried out before each measurement, and the equipment is calibrated every 2 to 3 months according to experience;

after the equipment is erected on the four-vision system, calibrating the four-vision system under a non-overlapping vision field through a corresponding high-precision coding three-dimensional target;

measuring unit stage:

when the chain runs, the four-eye vision system collects chain images according to a certain time interval, chains on two sides can be measured simultaneously, and then the center of a chain shaft in the images is automatically measured through elliptical detection based on deep learning, so that the distance between the chain shafts and the offset angle of the chain shafts are continuously and automatically calculated;

when the chain runs, the system acquires chain images according to a certain time interval and intelligently identifies, tracks and positions a chain shaft; secondly, when the tested chain is just in the central region of the image, extracting the center of a chain shaft;

the elliptic contour is directly fitted through an elliptic regression network based on the mask-CNN, so that the accuracy and the reliability of the extraction of the central pixel of the chain shaft are improved;

Mask R-CNN is a neural network for instance segmentation and object detection, which adds a segmentation head on the basis of Faster R-CNN for generating object masks; while the elliptic regression network is part of Mask R-CNN, it is used to predict the elliptic shape of the target instance;

the task of the elliptic regression network is to generate a minimum circumscribing ellipse (Minimum Bounding Ellipse) for the object instance after it is detected, for describing the shape and pose of the object; in Mask R-CNN, the elliptic regression network is performed on the basis of the output of the detection network, i.e. candidate boxes for the target instance have been obtained and the target class is predicted;

the elliptic regression network is a fully-connected neural network, takes a characteristic diagram output by the detection network as input, and outputs elliptic parameters of a target instance; specifically, it outputs the center coordinates, length of long and short axes, and rotation angle of the ellipse, and a confidence score for evaluating the accuracy and reliability of the ellipse;

the training of the elliptic regression network adopts a regression loss function, and the aim of the elliptic regression network is to minimize the difference between the predicted ellipse and the real ellipse; in practical applications, elliptic regression networks may be used for various tasks such as face detection, vehicle identification, ship detection, etc.

The structure of the elliptic regression network of the mask-CNN comprises a feature extraction network, an RPN module, an ROI Align module and an elliptic regression module which are sequentially carried out;

includes the steps of ResNet-FPN, RPN, feature map, roiAlign, refined features, conv, ellipse regression, which are performed sequentially.

The four-eye vision system calculates shaft center three-dimensional coordinates on two sides of the chain shaft respectively through two sets of binocular systems according to the obtained central pixel coordinates of the chain shaft and the calibration parameters;

the four cameras together form a multi-vision system under a non-overlapping vision field, three-dimensional target point information obtained by the binocular systems on two sides is unified to the same camera coordinate system, and a global coordinate system of the multi-vision system is obtained;

calculating the distance between the corresponding chains and the offset angle of the chain shaft;

early warning stage:

and the four-vision system generates a parameter report of the whole chain according to the monitoring result, and pre-warns the abnormal condition of the chain shaft through the server and the industrial personal computer.

The chain distance measuring method comprises a visual system calibration method without overlapping visual fields, which is used for calibrating visual systems with target information obtained by two visual systems not in the same coordinate system, carrying out joint calibration on all cameras, and jointly forming a multi-camera visual system by all cameras to obtain a reference camera coordinate system of the multi-camera system;

The method comprises the following steps:

s1, designing a three-dimensional target, wherein the size of the three-dimensional target is known, and the front surface and the back surface of the three-dimensional target are provided with target arrays consisting of coding targets with characteristic information; the coding target of the characteristic information comprises a positioning area and a coding identification area, wherein the positioning area is used for providing the position information of the target so as to realize the positioning of the target; the code identification area is used for providing code numerical value information;

in S1, a least square ellipse is selected to fit the center coordinates of the coding small circles of the coding mark points, and a single coding mark point area is identified and extracted according to the determined coordinates of the coding small circles of the coding mark points, so that the ID of the coding mark points is decoded according to the coding rule of the coding mark points;

s2, monocular calibration, wherein the calibration algorithm in S2 comprises the following steps: calibrating internal parameters of the camera and solving distortion coefficients to finish monocular calibration;

the monocular calibration in S2 includes coordinate conversion: performing camera internal reference solving by using a Zhang Zhengyou internal reference calibration method to obtain a radial distortion coefficient and a tangential distortion coefficient of the camera, and correcting projection coordinates by using the distortion parameters;

s2, changing the pose of the calibration plate at least once at single target timing, collecting the coded concentric circle array image, extracting the characteristic point coordinate pairs, substituting the characteristic point coordinate pairs into a camera model to calculate the initial internal and external parameters and distortion coefficients of the camera, and finishing single target calibration;

S3, multi-objective calibration, wherein the calibration algorithm in S3 comprises the following steps: solving relative external parameters between cameras, minimizing and optimizing the external parameters by utilizing the reprojection error, removing accumulated errors caused by matrix continuous multiplication, and completing the calibration of a multi-objective system under a non-overlapping view field;

the multi-objective in S3 is: calibrating each group of binocular cameras based on the internal and external parameters and the distortion coefficient of each camera obtained by the calibration of the monocular cameras to obtain a conversion relation between the coordinate systems of the cameras between the two groups of binocular cameras;

the coordinates of the multi-object targets in S3 are converted into: the method comprises the steps of obtaining a transformation matrix of a coordinate system between targets corresponding to two groups of binocular cameras through the geometric structure relation of the targets, obtaining the transformation matrix through solving a homography matrix or a projection matrix of a target photo, obtaining transformation matrices of all cameras with non-overlapping view fields through simultaneous solution, namely obtaining a transformation matrix from the coordinate system of all cameras to a global coordinate system, and completing four-dimensional system calibration.

The chain distance measuring method comprises a camera exposure time self-adaptive control method of line structure light image characteristics, and comprises a self-adaptive multi-exposure method;

the self-adaptive multi-exposure method is used for throwing auxiliary light to a shot object, wherein the light source of the auxiliary light is the most sensitive light source of a selected camera, and the shot object is subjected to multi-exposure to shoot an image of the auxiliary light;

The multiple exposure process records the following parameters: a confidence threshold T representing acceptable streak image quality; initial exposure time t _i The reliability of the gray level minimum position is larger than the threshold value T; the exposure time adjusting step length is delta t; the ratio K meeting the confidence threshold is a junctionA judgment standard of beam shooting;

the adaptive multi-exposure method comprises the following steps D1-D4:

d1, scanning the light bar image rows or columns at sampling intervals, and obtaining the light bar center point coordinates on each sampling row or column by adopting an adaptive threshold method;

d2, calculating gradients in the x and y directions of each pixel point in the neighborhood of the central point by using a Sobel gradient operator;

d3, calculating the normal direction of the center point by using a Bazon operator;

d4, calculating the light bar credibility based on a light bar credibility evaluation method by taking the coordinates of the central point and the normal direction vector as inputs, and calculating the gray level of the light bar;

gray scale intensity of the collected image is controlled by adjusting aperture of a lens or exposure time of a camera, a measuring area is obtained by each exposure, and finally three-dimensional data are restored by stripes of the measuring area and spliced;

the evaluation method of the light bar credibility comprises the following steps: taking the coordinates of the central point of a certain position of the light bar and the vector of the light bar in the normal direction of the position as inputs, and outputting the light bar credibility of the position;

The light bar credibility evaluation method comprises the steps of G1-G4:

the method comprises the steps of G1, estimating the width of a light bar section, taking the light bar section as a center, and searching pixel points with gray values which are not higher than 20% of the gray values of the center point towards two sides along the normal direction, wherein the pixel points are respectively used as a starting point and an ending point of the light bar section, and the sequence length between the two points is the width of the light bar section;

g2, fitting a Gaussian curve based on a light bar center point;

g3, calculating the light bar energy, namely the sum of gray values of all pixel points in the width range of the light bar section;

g4, calculating and correcting the light bar credibility through the light bar substrate noise;

the light bar credibility evaluation method comprises the steps S1-S4, and t is used for _i Exposing, calculating the reliability and gray scale of the light bar position,

s1, if the proportion of the position with the confidence coefficient larger than T is higher than K, ending, otherwise, reducing the exposure time to ensure that the confidence coefficient corresponding to the position with the maximum gray level of the light bar is larger than T;

s2, if the proportion of the position with the confidence coefficient larger than T is higher than K, ending, otherwise, increasing the exposure time to enable the confidence coefficient corresponding to the position with the minimum gray level of the light bar to be larger than T;

s3, if the proportion of the position with the reliability larger than T is higher than K, ending, otherwise, returning to S1.

The chain distance measuring method comprises an ellipse detecting method of Faster R-CNN, and comprises the following steps:

S1, acquiring a chain shaft picture, and labeling the picture to obtain a data set;

s2, sending the data set into a fast R-CNN network for training to obtain a training model;

the training process comprises a ResNet-FPN feature extraction module, an RPN module, a RoI Align module and an elliptic regression module;

s3, detecting and positioning the chain shaft ellipse by using a training model;

the ResNet-FPN feature extraction module is used for constructing a multi-scale feature pyramid FPN, extracting features at each level of the pyramid and predicting to finish multi-scale feature mapping;

the last layer of output of each stage is used as a characteristic, and the characteristics of each stage are fused to realize the detection of targets under different scales; PN module, after the chain shaft original image is subjected to characteristic extraction through the characteristic extraction network, obtaining a characteristic image, and then carrying out region extraction through RPN module;

the RPN adds three convolution layers on the feature map to perform region selection, wherein the first convolution layer is used for adjusting the channel number of the feature layer, the second convolution layer is used for scoring the foreground and the background of the anchor box generated by each anchor point, and the third convolution layer is used for performing frame regression by generating the frame regression weight of the anchor box for each anchor point;

the RoI Align module is a regional feature aggregation mode, the layer is based on an extended prediction region, and an image value on a pixel point with a floating point number is obtained by using a bilinear interpolation method, so that the whole feature aggregation process is converted into a continuous operation, a small feature map is selected again, the extracted features are accurately aligned with the input of the FPN, and the accuracy of the predicted ellipse parameters is adversely affected due to the extended features outside the prediction region;

Zero filling is carried out on the extended feature region subjected to RoI alignment, and the zero filling is used for refining the feature region;

the elliptic regression module is introduced on the basis of bounding box regression and is used for identifying and positioning the position of the edge of the elliptic contour;

the Faster R-CNN basic model is of a front-end network structure, and regression of five relative offset parameters of the ellipse realizes regression of the profile of the chain shaft; the relative offset parameters of the prediction boundary boxes normalize the five parameters of the ellipse boxes, so that objects with different sizes in the image can equally contribute to regression loss.

The chain distance measuring method comprises a non-overlapping view field multi-camera combined calibration method, and comprises the following steps of:

s1, monocular calibration is carried out on cameras by using a target calibration plate to obtain internal parameters, external parameters and distortion coefficients of each camera; adjusting internal parameters and external parameters of the camera by using a binding algorithm, and distortion coefficients; the target calibration plate is provided with a specific pattern, the characteristic pattern comprises a circular graph and an annular graph which is surrounded by the circular graph, the two patterns are different in color, the annular graph is provided with color points, and the color points are a dot area graph; shooting an image of the target calibration plate by a camera, so that the target calibration plate is completely arranged in the image acquired by the field of view; solving the pose among all cameras by continuously moving the target calibration plate;

S2, calibrating parameters of adjacent cameras;

s3, the target calibration plate is arranged under the field of view of the adjacent camera to calibrate the camera, and the target calibration plate is moved to realize the parameter calibration of the multi-camera through pose conversion; the binding algorithm adjusts and optimizes the pose relation of the camera;

the camera imaging model is:

wherein the method comprises the steps of，f _x ，f _y ，u ₀ ，v ₀ Is a camera internal parameter, R, t is a camera external parameter, (X) _W ，Y _W ，Z _W ) Is the coordinate under the world coordinate system;

the camera perspective imaging model is as follows:

k is a camera internal parameter matrix, s is a scale factor, and H is a homography matrix;

the rotation vectors r1 and r2 are in unit orthogonal relationship, that is:

r ₁ ^T r ₂ ＝0

r ₁ ^T r ₁ ＝r ₂ ^T r ₂ ＝1

the method can obtain:

r ₁ ＝K ^-1 H ₁

r ₂ ＝K ^-1 H ₂

substituting the camera imaging model can obtain:

obtaining initial values of an internal parameter K, an external parameter and a distortion coefficient of the camera by utilizing an SVD algorithm;

calculating the accurate value of the camera parameter in a mode of minimizing the system re-projection error by using a binding adjustment optimization method; reprojection

The error formula is:

wherein m is _ij Is the two-dimensional observation coordinate of the target point,the projection coordinates are, P is the three-dimensional coordinates of the target point, K is the camera internal reference matrix, R, t is the camera internal reference, and K ₂ ,p ₁ ,p ₂ ,k ₃ Is a distortion coefficient;

converting into unconstrained plum cluster solution, calculating a Jacobian matrix, and obtaining a result through multiple iterations by using a Levenberg-Marquardt algorithm;

The calculation formula of the binocular camera calibration is as follows:

wherein, R1, T1 and R2, T2 respectively represent the transformation matrix from the world coordinate system to the left and right camera coordinate systems, R, T represent the transformation matrix from the left camera coordinate system to the right camera coordinate system;

deducing camera coordinates of three-dimensional coordinates of target points under the view field of the right camera in the coordinate system of the left camera according to the fixed position relation between the target patterns, and obtaining the three-dimensional coordinates of the points under the coordinate system of the right camera according to the calculated conversion matrix between the two cameras;

calculating an error between the camera internal parameter and a real projection value according to the pixel coordinate of the internal parameter re-projection of the camera;

repeating the steps to obtain the re-projection errors of all cameras, fixing the internal references of the cameras, and obtaining a result by using a BA algorithm, wherein the result comprises a conversion matrix between the cameras and the real coordinates of the three-dimensional points of the targets;

the pose relation between cameras is improved as follows:

wherein u is _i 、v _i Representing the two-dimensional coordinates of the target point on the real image,representing two-dimensional coordinates of target points re-projected to a pixel coordinate system according to camera parameters, and n represents the sum of the numbers of all target points.

The application has the following beneficial effects:

the application realizes the calibration and measurement under the four-eye vision system, obtains the distance and the chain shaft offset angle of the corresponding chain, realizes the on-line monitoring of the three-eye and above multi-eye vision system, and has the characteristics of safety, high efficiency and real time;

Two sets of machine vision sensor systems are erected on two sides of a chain, each set of system comprises a group of binocular vision systems formed by two industrial cameras, four cameras jointly form a four-eye camera vision system under a non-overlapping visual field, and simultaneously, lasers, light sources and the like are used for providing additional three-dimensional information or providing peripheral peripherals of a good image acquisition environment, so that the calculation of the three-dimensional coordinates of the center of a chain shaft is realized;

the transmission chain spacing four-vision system is mainly divided into a calibration part and a measurement part; calibrating a four-eye vision system under a non-overlapping visual field through a corresponding high-precision coding three-dimensional target; when the chain runs, the system collects chain images according to a certain time interval, chains on two sides can be measured simultaneously, and then the center of a chain shaft in the images is automatically measured through elliptical detection based on deep learning, so that the distance between the chain shafts and the offset angle of the chain shafts are continuously and automatically calculated.

Drawings

FIG. 1 is a schematic diagram of a system of the present invention;

FIG. 2 is a flow chart of the present invention;

FIG. 3 is a diagram of an elliptic regression network based on Mask R-CNN according to the present invention;

FIG. 4 is a schematic diagram of the distance between the axes of the chains and the angle of the offset of the axes according to the present invention;

FIG. 5 is a system flow diagram of a vision system calibration method of the present invention without overlapping fields of view;

FIG. 6 is a schematic view of a three-dimensional coded target of the vision system calibration method without overlapping fields of view of the present invention;

FIG. 7 is a schematic diagram of the spatial relationship of the coordinate system of the vision system calibration method without overlapping fields of view of the present invention;

FIG. 8 is a schematic illustration of multi-targeting based on stereo coded targets under non-overlapping fields of view of the present invention;

FIG. 9 is a concentric circle calibration plate with encoding function of the vision system calibration method of the present invention without overlapping fields of view;

FIG. 10 is a schematic drawing of the extraction of calibration plate feature points for the vision system calibration method without overlapping fields of view of the present invention;

FIG. 11 is a system coordinate system conversion relationship of the vision system calibration method of the present invention without overlapping fields of view;

FIG. 12 is a flow chart of an adaptive multiple exposure method of the present invention;

FIG. 13 is a graph of the gray scale statistics of the cross section of an image light bar acquired by the adaptive multi-exposure method of the present invention;

FIG. 14 is a chart b of gray scale statistics of cross sections of light bars of images collected by the adaptive multi-exposure method of the present invention;

FIG. 15 is a chart of gray scale statistics c of cross sections of light bars of images collected by the adaptive multi-exposure method of the present invention;

FIG. 16 is a diagram of a chain axis ellipse detection network of the ellipse detection method of Faster R-CNN of the present invention;

FIG. 17 is a schematic diagram of the effect of a chain-axis ellipse detection network of the ellipse detection method of Faster R-CNN of the present invention;

FIG. 18 is a schematic diagram of the elliptic regression parameter drift of the method for elliptic detection of Faster R-CNN according to the present invention;

fig. 19 is a structure diagram of a StyleGAN3 generator of the pin image data enhancement method of the present invention;

FIG. 20 is a block diagram of a Projector discriminator of the pin image data enhancement method of the invention;

FIG. 21 is a CCM block diagram of the pin image data enhancement method of the present invention;

FIG. 22 is a CSM block diagram of a pin image data enhancement method of the present invention;

FIG. 23 is a flowchart of a pin image generation method according to the present invention;

FIG. 24 is an image generation effect diagram of a pin according to the present invention;

FIG. 25 is a flow chart of a non-overlapping field-of-view multi-camera joint calibration method of the present invention;

FIG. 26 is a diagram of a target calibration plate of the non-overlapping field-of-view multi-camera joint calibration method of the present invention;

FIG. 27 is a graph of target calibration plate recognition effect of the non-overlapping field-of-view multi-camera joint calibration method of the present invention;

FIG. 28 is a schematic diagram of a non-overlapping field-of-view multi-camera system of the non-overlapping field-of-view multi-camera joint calibration method of the present invention;

in the figure:

1. blowing device

2. Industrial hairbrush strip module

3. Industrial control computer

4. Temperature sensor

5. Humidity sensor

6. Atmospheric pressure sensor

7. Light source A

8. Camera A

9. Camera B

10. Light source B

11. Light source C

12. Camera C

13. Camera D

14. A light source D;

Detailed Description

The following describes the embodiments of the present invention in further detail with reference to the accompanying drawings. These embodiments are merely illustrative of the present invention and are not intended to be limiting.

In the description of the present invention, it should be noted that the terms "center", "longitudinal", "lateral", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, are merely for convenience in describing the present invention and simplifying the description, and do not indicate or imply that the apparatus or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.

In the description of the present invention, it should be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to the specific circumstances.

Furthermore, in the description of the present invention, unless otherwise indicated, the meaning of "a plurality" is two or more.

Examples

As shown in fig. 1, a schematic diagram of a chain monitoring system is provided, two sets of machine vision sensor systems are erected at about 0.6 m positions on two sides of a chain, each set of system comprises a group of binocular vision systems consisting of two industrial cameras, the four cameras together form a four-eye camera vision system under a non-overlapping visual field, and meanwhile, a laser, a light source and the like are used for providing additional three-dimensional information or peripheral peripherals providing a good image acquisition environment for calculating the three-dimensional coordinates of the center of a chain shaft.

As shown in fig. 1, a blowing device 1, an industrial hairbrush module 2, an industrial personal computer 3, a temperature sensor 4, a humidity sensor 5, an atmospheric pressure sensor 6, a light source A7, a camera A8, a camera B9, a light source B10, a light source C11, a camera C12, a camera D13 and a light source D14 are arranged on two sides of a chain;

the four cameras of the four-vision system are a camera A8, a camera B9, a camera C12 and a camera D13;

the set light sources are a light source A7, a light source B10, a light source C11 and a light source D14;

the four-eye system of the whole transmission chain interval is shown in figure 1, and a plurality of devices are discontinuously arranged at the positions along the line which do not influence the movement of the chain; firstly, two cameras are placed on a special tripod and fixed by the tripod below, and the whole height of the tripod can be freely telescopic and adjustable to form a mobile portable binocular system; in order to meet the system requirement of simultaneously measuring the chain shafts on two sides of the chain, a set of binocular system is respectively erected on two sides of the chain shaft, and four cameras together form a four-eye system, which is also a core component of the whole system; secondly, because the field working condition is complex and dim, in order to overcome the influence of the working environment on the imaging quality, a light source is additionally arranged near the annular area outside the camera; meanwhile, in order to solve the influence of dust on the chain image in the working environment, the blowing equipment 1 is adopted and the industrial hairbrush strip module 2 is installed, and the chain shaft is purged and cleaned before the image is acquired by a camera, so that the imaging quality of the camera is ensured.

The whole flow chart of the transmission chain spacing four-vision system is shown in fig. 2, and is mainly divided into a calibration part and a measurement part; firstly, after equipment is erected on a system, calibrating a four-eye vision system under a non-overlapping visual field through a corresponding high-precision coding three-dimensional target; secondly, when the chain runs, the system collects chain images according to a certain time interval, chains on two sides can be measured simultaneously, and then the center of a chain shaft in the images is automatically measured through elliptical detection based on deep learning, so that the distance between the chain shafts and the offset angle of the chain shafts are continuously and automatically calculated; finally, the system generates a parameter report of the whole chain according to the monitoring result, and the abnormal condition of the chain shaft is pre-warned through the server and the industrial personal computer 3.

The system is convenient and simple to operate, and the specific steps of the system are as follows: after the equipment is erected, an operator holds the corresponding three-dimensional coding target and acquires 15-20 images according to the requirement at a position about 0.6 meter away from the equipment camera, in the process, the system automatically guides the operator how to acquire the images, and further the system automatically completes the calibration of a four-eye vision system under a non-overlapping visual field according to the acquired target images; after the calibration process is familiar, the time for drawing and calibrating the whole system is less than 5 minutes, and the process can be completed by a single person only; after the calibration is finished, the accuracy of the equipped high-accuracy target rod automatic detection system can be used; the calibration process is not required to be carried out before each measurement, and the equipment is calibrated every 2 to 3 months according to experience.

The system measurement part comprises the following specific steps: firstly, when a chain runs, a system collects chain images according to a certain time interval and intelligently identifies, tracks and positions a chain shaft; secondly, when the tested chain is just in the central region of the image, extracting the center of a chain shaft; in order to increase the accuracy and reliability of the extraction of the central pixel of the chain shaft, the elliptic outline is directly fitted through an elliptic regression network based on Mask R-CNN, and the network structure diagram is shown in figure 3 and can be divided into four main parts, namely a characteristic extraction network, an RPN module, an ROI alignment module and an elliptic regression module; then, the system calculates the three-dimensional coordinates of the shaft center at two sides of the chain shaft through two sets of binocular systems according to the obtained coordinates of the pixel at the center of the chain shaft and the calibration parameters of the system; the four cameras together form a multi-vision system under a non-overlapping vision field, so that three-dimensional target point information obtained by the binocular systems on two sides is unified to the same camera coordinate system, namely a global coordinate system of the multi-vision system; at the moment, the distance and the chain shaft offset angle of the corresponding chains can be calculated, and the schematic diagram of the distance and the chain shaft offset angle is shown in fig. 4; finally, the system can send corresponding chain parameter reports and early warning information through the server and the industrial personal computer 3 according to the monitoring result.

The chain distance measuring method comprises the following visual system calibration method without overlapping view fields, wherein the target to be measured is the chain shafts at the two sides of the chain, and the target cannot be simultaneously appeared under the same camera view field, so that a group of rigidly connected binocular cameras are respectively erected at the two sides of the chain, and are used for respectively carrying out 3D position measurement on the chain shafts at the two sides; the two groups of binocular cameras do not have overlapping view fields, and the central information of chain shafts at two sides cannot be unified under the same camera coordinate system; therefore, based on the system measurement requirement of the non-overlapping view field, the three-dimensional target is designed to jointly calibrate the four cameras, so that the four cameras jointly form a four-eye camera vision system, and the chain shaft center information on two sides is unified under the same camera coordinate system, namely under the reference camera coordinate system of the multi-eye system, and the overall flow chart of the system is shown in fig. 5;

the size of the adopted three-dimensional target is precisely known, and the front side and the back side are not traditional checkerboards, but are target arrays formed by coding targets with characteristic information, as shown in fig. 6; the design principle of the single target is as follows: the target comprises a positioning area and a coding recognition area, wherein the positioning area is used for providing position information of the target so as to realize positioning of the target; the code identification area is used for providing code numerical value information; selecting a least square ellipse fitting coding small circle center coordinate of a coding mark point, identifying and extracting a single coding mark point region according to the determined coding mark point coding small circle center coordinate, and decoding the ID of the coding mark point according to a coding rule of the coding mark point;

The calibration algorithm mainly comprises two parts, namely, firstly, calibrating internal parameters of a camera and solving distortion coefficients, namely, monocular calibration is carried out; secondly, solving relative external parameters between cameras, optimizing the external parameters by utilizing the minimization of the reprojection error, and avoiding the accumulated error caused by matrix continuous multiplication, namely, performing multi-objective system calibration under a non-overlapping visual field;

the coordinate conversion related to monocular calibration is shown in fig. 7, specifically, a Zhang Zhengyou internal reference calibration method is adopted to carry out internal reference solution of a camera, the method overcomes the defect that the traditional calibration method has high requirement on the precision of a calibration object, and meanwhile, the problem of poor robustness of the self-calibration method is solved; taking the influence of radial distortion and tangential distortion on an image into consideration, obtaining a radial distortion coefficient and a tangential distortion coefficient of a camera, and correcting projection coordinates by using distortion parameters; when in calibration, the pose of the calibration plate is changed for a plurality of times, the coded concentric circle array image is acquired, the characteristic point coordinate pairs are extracted, and the characteristic point coordinate pairs are substituted into a camera model to calculate the initial internal and external parameters and distortion coefficients of the camera, so that monocular calibration is completed;

as shown in fig. 11, four cameras respectively establish a camera coordinate systemTo the point ofPixel coordinate system +. >To->Since the four cameras are mounted at known fixed positions on the camera frame, the rotation matrix R between their coordinate systems can be derived by calibration ₁₂ 、R ₃₄ And translation vector t ₁₂ 、t ₃₄ ；

Two columns in FIG. 11 are equivalent diagrams of chain shaft, and we provide two sets of binocular cameras for acquiring circle centers P of bottom surfaces of two sides of chain shaft ₁ (X _W1 ,Y _W1 ,Z _W1 )、P ₂ (X _W2 ,Y _W2 ,Z _W2 )、P ₃ (X _W3 ,Y _W3 ,Z _W3 )、P ₄ (X _W4 ,Y _W4 ,Z _W4 ) And the pixel coordinates projected into the image. With P ₁ (X _W1 ,Y _W1 ,Z _W1 ) For example, its projected pixel coordinates in camera 1 areThe projected pixel coordinates in camera 2 are +.>

In order to improve and repeat the calibration, by using the principle, a multi-view system formed by four cameras is shown in fig. 8 by means of the three-dimensional coding targets shown in fig. 6, 9 and 10, wherein two adjacent cameras C1 and C2 are two-view cameras with rigid connection and overlapped view fields, the other two adjacent cameras C3 and C4 are another group of two-view cameras, the optical axis directions of the two groups of two-view cameras are opposite, and due to the shielding of an object to be detected in the field working condition, the two groups of two-view cameras have no common view field or have a very small common view field; the pose relation between the two groups of binocular cameras can be obtained by calibrating the common view field between the binocular cameras, and then the two groups of binocular systems are combined by utilizing the geometrical relation of the three-dimensional target, so that the calibration of the multi-view system under the non-overlapping view field is completed; the specific operation of the multi-objective calibration is that firstly, based on the internal and external parameters and distortion coefficients of each camera obtained by the single-objective camera calibration, two groups of double-objective cameras can be calibrated respectively to obtain the conversion relationship between the camera 1 coordinate system and the camera 2 coordinate system and the conversion relationship between the camera 3 coordinate system and the camera 4 coordinate system; secondly, a transformation matrix from a target 2 coordinate system to a target 1 coordinate system can be obtained according to the geometric structure relation of the target; the transformation matrix of the target 1 to the camera 1 coordinate system and the transformation matrix of the target 2 to the camera 3 coordinate system can be obtained by carrying out homography matrix solving when the target is photographed, so that the transformation matrix of the camera 3 coordinate system with non-overlapping fields of view to the camera 1 coordinate system (namely, the global coordinate system) can be obtained through simultaneous solution, and the four-eye system calibration is completed.

Monocular camera calibration:

in the chain spacing monitoring system, cameras are required to acquire parameter matrixes in the cameras through monocular calibration respectively, and distortion coefficients of the cameras are acquired simultaneously; the calibration plate for camera calibration is shown in figures 9 and 10,

converting an image point (x, y) on an imaging surface into an image surface point (u, v) using an in-camera parametric model, having:

wherein (u) ₀ ,v ₀ ) A is the intersection point of the optical axis center and the imaging surface _x And a _y Proportional to X-axis and Y-axisLarge coefficients. Obtaining:

when f _x ＝f _y When =f, the model contains 3 parameters (f, u ₀ ,v ₀ ) Considering f in practical application _x And f _y The internal parameter matrix contains 4 parameters (f _x ,f _y ,u ₀ ,v ₀ ) And (3) expanding the formula (3.2), selecting coordinates of one point on the object, substituting the coordinates, and calculating to obtain an internal reference matrix K of the camera.

Considering the influence of the distortion of the lens on the image, the radial distortion coefficient and tangential distortion coefficient of the camera are also required to be obtained. The correction formula for the projection coordinates by using the distortion parameters is as follows:

wherein: (x) _d ,y _d ) Image coordinates of the imaging plane distortion point; (x) _p ,y _p ) Correcting the image coordinates of the points for the normalized imaging plane;k ₁ ，k ₂ and k ₃ Is the radial distortion coefficient of the lens; p is p ₁ And p ₂ Is the tangential distortion coefficient of the lens.

Binocular camera calibration

The external parameter calibration between two cameras of the binocular system is to convert coordinate points in a right camera coordinate system into a left camera coordinate system through rotation and translation (taking the left camera coordinate system as a base coordinate system). The rotation and translation conversion matrix can be formed by combining a rotation matrix and a translation vector, images on the same calibration plate are acquired through two cameras, and external parameters between two camera coordinate systems can be obtained by utilizing characteristic point coordinates under the calibration plate coordinate systems.

A certain part on the calibration plateThe three-dimensional points of each corner point under the coordinate system of the left camera, the coordinate system of the right camera and the world coordinate system where the calibration plate is positioned are respectively P _a (X _a ,Y _a ,Z _a )、P _b (X _b ,Y _b ,Z _b ) P _w (X _w ,Y _w ,Z _w ) The following steps are:

the relative relation between the left camera and the right camera in binocular stereoscopic vision can be obtained by the above method:

P _a ＝R ₁₂ P _b +t ₁₂

wherein [ R ] ₁₂ t ₁₂ ]Representing a rotational translation matrix from the right camera coordinate system to the left camera coordinate system:

from the dual-object determination, the final solution is to obtain a rotation translation matrix [ R ] between the side camera coordinate system and the front camera coordinate system ₁₂ t ₁₂ ]。

The invention relates to a chain distance measuring method, which comprises a camera exposure time self-adaptive control method of line structure light image characteristics;

based on improvement of the existing light stripe credibility evaluation method:

taking the coordinates of the central point of a certain position of the light bar and the vector of the light bar in the normal direction of the position as inputs, and outputting the light bar credibility of the position; the algorithm mainly comprises four parts:

1. estimating the width of the light bar section, starting from a central point, searching pixel points with gray values which are not higher than 20% of the gray values of the central point for the first occurrence to two sides along the normal direction by taking the central point as the center, and respectively taking the pixel points as a starting point and an ending point of the light bar section, wherein the sequence length between the two points is the width of the light bar section;

2. Fitting a Gaussian curve based on the center point of the light bar;

3. calculating the energy of the light bar, namely the sum of gray values of all pixel points in the width range of the section of the light bar;

4. and calculating the corrected light bar credibility through the light bar substrate noise.

The self-adaptive multi-exposure method is adopted, and the operation is as follows:

1. and (3) putting auxiliary light into the shot object, wherein the light source is the most sensitive light source of the selected camera, shooting the image of the auxiliary light by performing multiple subsequent exposures (controlling the gray level intensity of the acquired image by adjusting the aperture of the lens or the exposure time of the camera, ensuring that at least one area is suitable for measurement in each exposure, and finally restoring three-dimensional data by stripes of the suitable area to be spliced.

2. The multiple exposure process records the following parameters:

(1) A confidence threshold T representing acceptable streak image quality;

(2) Initial exposure time t _i The reliability of the gray level minimum position is larger than the threshold value T;

(3) The exposure time adjusting step length is delta t;

(4) The minimum proportion K of the confidence threshold is satisfied, which is a criterion for ending shooting.

The specific algorithm flow is as follows: firstly, scanning light bar image lines (or columns) according to a certain sampling interval, and obtaining the center point coordinates of the light bars in each sampling line (or column) by adopting an adaptive threshold method; then, for each pixel point in the neighborhood of the center point, calculating the gradient in the x and y directions by using a Sobel gradient operator, and then calculating the normal direction of the center point by using a Bazon operator; and finally, taking the coordinates of the central point and the normal direction vector as input, calculating the light bar credibility based on a light bar credibility evaluation method, and calculating the light bar gray level.

As shown in the flow chart of the adaptive multi-exposure method in fig. 12, the light stripe reliability evaluation method comprises: at t _i Exposing, calculating the reliability and gray scale of the light bar position,

s3, if the proportion of the position with the reliability larger than T is higher than K, ending, otherwise, returning to S1;

the self-adaptive control method for the exposure time of the line-structured light sensor mainly comprises three processes:

(1) Scanning the image according to rows/columns, and determining the coordinates of the central point of the light bar;

(2) Calculating the normal direction of the center point;

(3) And evaluating the reliability of the light bars, and calculating the gray scale.

The gray level statistical diagrams of the light bar section of the camera exposure time self-adaptive control method adopting the line structure light image characteristics are shown in fig. 13, 14 and 15, and the gray level statistical model of the light bar is close to an ideal state, so that the effectiveness of the method is proved.

The invention relates to a chain distance measuring method, which comprises the following steps of:

The ellipse detection adopts a neural network target detection method based on CNN, a large number of chain shaft pictures are collected, manual labeling is carried out, a data set is manufactured, the data set is sent into a Faster R-CNN network for training, a training model is obtained, the training model is utilized to realize ellipse detection and positioning tasks for the existing chain shafts, and the flow is shown in a figure 16;

as shown in the elliptic regression network structure based on the fast-CNN architecture of fig. 16, it can be divided into 4 parts, namely a feature extraction network, an RPN module, an ROI alignment module and an elliptic regression module;

as shown in fig. 16, the flow mainly includes calculation of four modules;

a ResNet-FPN feature extraction module; because ResNet only uses the top-most characteristic of the image to predict, the phenomenon that the accuracy is not high can appear in small target detection, the method for solving this problem is to construct a multi-scale characteristic pyramid FPN, extract the characteristic at each level of the pyramid and predict; an FPN structure is established on the ResNet to complete a multi-scale feature mapping function, the last layer of output of each stage is used as a feature, and the features of each stage are fused to realize detection of targets under different scales; the FPN is the forward calculation of CNN from bottom to top, and is composed of multi-scale feature mapping, the last layer of output of each stage is used as a feature, pyramid is created, the deeper stage features are stronger, and when targets with different scales are detected, the method can obtain a feature map with proper scales and strong semantic information;

An RPN module; the method comprises the steps that after a chain shaft original image is subjected to feature extraction through a feature extraction network, a feature image is obtained, and then region extraction is performed through an RPN module; the method is characterized in that three convolution layers are added to a feature map for region selection by an RPN, the first convolution layer is used for adjusting the channel number of the feature layer, the second convolution layer is used for scoring the foreground and the background of an anchor box generated by each anchor point, and the third convolution layer is used for generating the frame regression weight of the anchor box for each anchor point to carry out frame regression;

a RoI Align module; the method is characterized in that the layer is based on an extended prediction area, and an image value on a pixel point with a floating point number as a coordinate is obtained by using a bilinear interpolation method, so that the whole feature aggregation process is converted into a continuous operation, a small feature map is selected again, extracted features are accurately aligned with the input of the FPN, and because the extended features outside the prediction area have adverse effects on the precision of the predicted ellipse parameters, zero filling is needed to be carried out on the extended feature area passing through the RoI alignment to refine the feature area;

an ellipse regression module; in order to accurately identify and position the edge of the elliptical contour, an elliptical regression module is introduced on the basis of bounding box regression; adopting a Faster R-CNN basic model as a front-end network structure; adopting a method similar to that of the Faster R-CNN regression frame to predict offset parameters for chain shaft profile regression, and directly regressing five relative offset parameters of ellipse; the method for regression of the boundary frames by using the relative deviation parameters of the prediction boundary frames has the advantages that all five parameters of the ellipse frames are normalized, and objects with different sizes in the image can equally contribute to regression loss, so that the loss is not influenced by the size of the object; in addition, normalization ensures that when the fitting area approaches to a true value, all predicted offset values approach to zero, so that the training process tends to be stable, and the situation of no boundary value is avoided;

The method for regression of the boundary frames by using the relative deviation parameters of the prediction boundary frames has the advantages that all five parameters of the ellipse frames are normalized, and objects with different sizes in the image can equally contribute to regression loss, so that the loss is not influenced by the size of the object; in addition, normalization ensures that when the fitting area approaches to a true value, all predicted offset values approach to zero, so that the training process tends to be stable, and the situation of no boundary value is avoided;

geometrically, a general ellipse with arbitrary direction can be defined by its five parameters: center coordinates (x) ₀ ,y ₀ ) The semi-major and semi-minor axes a, b, the rotation angle θ (from the positive horizontal axis to the major axis of the ellipse), give the standard form of a general ellipse as follows:

x′＝x-x ₀ ,y′＝y-y ₀

wherein the rotation angle θ ε (-pi/2, pi/2)]Carrying out regression prediction on five elliptic parameters from the characteristic region to an elliptic true value through a complete regression training; given a square feature region q= (Q) extended by a proposal P _x ,Q _y ,Q _l ) The goal is to learn a set of relative offset parameters between the regression feature Q and the elliptical true value E, here we use delta _x ，δ _y ，δ _a ，δ _b ，δ _θ Five parameters parameterize the regression process.

True elliptic radius range is E _θ ∈(-π/2,π/2]，δ ^* E' is a predicted ellipse calculated from δ, which is a regression target of the ellipse, (δ _x ,δ _y ) Scale invariant translation, δ, of specified Q to E _a And delta _b Logarithmic spatial translation of specified Q to E' semi-major and semi-minor axes, delta _θ Is an E' normalized ellipse angular direction prediction, when the predicted region (p→q) approaches the true ellipse E, the predicted offset δ is bounded, and the regression process is as in fig. 18;

the chain shaft ellipse detection method realized by the invention comprises three processes:

(1) Acquiring and preprocessing a chain shaft image;

(2) Inputting the image into a network model for detection;

(3) Performing ellipse fitting on the detection parameters and performing real-time image positioning; the specific flow is as follows: acquiring an image to calibrate a data set, constructing a network model based on a pytorch frame, and performing network training on the data set; the network offline prediction adopts an image of a chain joint acquired in real time, a mapping from an input image to an output recognition result is obtained through direct learning of a series of feature mapping transformations, and the elliptic contour information of a chain shaft end point is output; by tracking and positioning the chain shaft end circle in real time, the network outputs elliptic parameters and the chain shaft end circle outline to mark the position of the extracted chain shaft end circle; the network detection effect is shown in fig. 17; from the perspective of the elliptic effect, the feasibility of the method based on the Faster R-CNN method is verified.

The chain spacing measuring method of the invention comprises a pin shaft image data enhancing method,

first an image dataset with pins is made, the model training goal is to be able to generate high quality images with multiple transformations. In order to improve the training efficiency of the network, the network structure is adjusted, and the model training strategy is changed, so that the training time and the calculation cost are greatly reduced, and the sample diversity is rapidly realized;

the technical proposal is as follows:

1) Original dataset production and annotation

The premise of completing the displacement change real-time monitoring work between the chain shafts based on the deep learning elliptic parameter regression model is that a large number of images of the chain pin shafts are collected in advance, wherein the images comprise a certain amount of images with different angles and orientations, so that the production and labeling of a data set are vital to the displacement change real-time monitoring work between the chain shafts; the data set images are all acquired from actual engineering projects by the inventor, and the binocular industrial camera built by the system is utilized for acquiring the images of the pin shafts; the original data set has low requirements on the number of pin shaft images, but needs to comprise images of various angles; in the aspect of data set labeling, the flow is mainly divided into three: data cleaning, data labeling and labeling inspection; in order to obtain higher image labeling quality, the dataset labeling tool adopts autonomously developed labeling software to label datasets;

2) Model-generating network construction and training optimization

Because of the characteristics of high similarity and unobvious characteristics of the pin images, the generated model is required to achieve higher generation quality; therefore, the invention builds a neural network model by improving a related structure based on the latest generation model StyleGAN-XL, thereby realizing the rapid and high-quality synthesis of the pin shaft image;

the main factor of the performance degradation of the generated model StyleGAN on some large unstructured data sets is its training strategy, because the StyleGAN design is controllable, its limiting design is not suitable for different data sets, especially on industrial detection small scene data sets; styleGAN-XL can successfully train the latest StyleGAN3 generator on a high resolution image dataset using neural network priors and a progressive growth strategy according to the recently proposed ProjectedGAN paradigm;

specifically, our goal is to successfully train a StyleGAN3 generator on the pin image dataset; since training becomes unstable at high resolution, resulting in higher FID, adjustments are made as follows; firstly, modifying a generator and regularization loss thereof, and adjusting potential space to be suitable for configuration of ProjectedGAN and class conditions; then reintroducing progressive growth (pixel increment) to increase training speed and performance; finally, a plurality of feature networks are utilized to find a feature network suitable for ProjectedGAN training;

The StyleGAN3 generator consists of a mapping network Gm and a synthesis network Gs; the network structure is shown in fig. 19; first, gm maps a normally distributed potential code z to a pattern code w; the pattern code w is then used to modulate the convolution kernel of Gs to control the synthesis process; the synthetic network Gs of StyleGAN3 starts with a spatial mapping defined by fourier features; this input is then convolved, non-linearly and upsampled by the N layers to generate an image; each non-linearity is wrapped by up-sampling and down-sampling operations to prevent aliasing; the low pass filters used for these operations are carefully designed to balance image quality, anti-aliasing, and training speed; the number of layers N is 14, and is irrelevant to the final output resolution; the progressive growth strategy is considered for stable training under high resolution to improve the model convergence speed and the image synthesis quality;

ProjectedGAN expands the original resistance game between the generator G and the discriminator D by a set of characteristic Projectors { Pl }; the Projector maps the images generated by the real images x and G to the input space of the discriminator. The network structure is shown in fig. 20. ProjectedGAN targets are expressed as:

where { Dl } is a set of independent discriminators that handle different feature Projectors; the Projector consists of a pre-trained feature network F, cross-channel blending (CCM) and cross-scale blending (CSM) layers, and the specific structure is shown in fig. 21 and 22;

The purpose of CCM and CSM is to prohibit the discriminator from focusing on only a subset of its input feature space, resulting in a mode crash; both modules employ a microstochastic projection that is not optimized during GAN training; CCM mixes the characteristic of the cross-channel through the random 1x1 convolution, CSM mixes the characteristic of the cross-scale through the residual random 3x3 convolution block and bilinear upsampling; the output of CSM is a feature pyramid composed of four feature graphs with different resolutions; four discriminators operate independently on these feature maps; each discriminator uses a simple convolution structure and spectral normalization; the depth of the discriminator depends on the resolution of its input, i.e. a spatially larger feature map corresponds to a deeper discriminator; apart from spectral normalization, the Projected GAN does not use additional regularization, such as gradient penalty; finally, the application of a micro-data enhancement before F improves the performance of the proposed GAN independent of the data set size;

the whole working flow of the model is as follows: first, providing a pre-trained potential code z and a class label c to a pre-trained embedding and mapping network Gm to generate a pattern code w; these codes modulate the convolution of the composite network Gs; in the training process, the layers are gradually increased, so that the output resolution of each stage of the step-growth plan is doubled; only the latest layer is trained, and the fixed state of other layers is kept; when less than 2242, up-sampling the synthesized image and passing through CNN (convolutional neural network), viT (Vision Transformer model) and respective feature mixture blocks (ccm+csm); at higher resolution, CNN receives unchanged images, while ViT receives a downsampled input to keep memory requirements low, but still utilize its global feedback; finally, we apply 8 independent discriminators on the resulting multi-scale feature map; the image is also provided to a classifier CLF for classifier guidance;

Training StyleGAN-XL on an original pin shaft image data set, and cutting and compressing the data set image according to different resolutions according to the requirement according to a progressive growth training strategy; then, continuously adjusting the initial resolution to achieve the effect of generating a real image contour, and taking the pre-training model under the resolution as a source model for subsequent high-resolution training; in the whole training time, the training efficiency of the model is ensured by properly adjusting the data set according to the size of the data set;

setting a stylegan3-t 10-layer network with a resolution of 642 to start progressive growth training, cutting off 2 layers each time the resolution increases, and adding 7 new layers; if the number of layers is small, the model can have poorer performance; if too much is added, the cost is increased, and the benefit is reduced; at the last 10242 resolution, we only add 5 layers since the last two layers are not discarded; each phase is not a fixed growth plan, but rather is trained so that FID is not decreasing; because of hardware, the batch processing size of the model cannot be tested, smaller batch size is uniformly adopted, once a new layer is added, the low-resolution layer is kept unchanged, so that the mode is prevented from crashing, and then the model under each resolution is utilized for image generation;

In the aspect of model training, the number of network layers and the number of training images are reasonably set through configuration parameters, so that the complexity of a model is further reduced, and the model training time is shortened; compared with other existing generation models, the method has the following advantages:

the pin shaft image generation model based on StyleGAN-XL has the advantages of simple network structure, easy training, high image generation quality and the like; according to the invention, by introducing a high-efficiency generation model, a relatively high image generation quality can be obtained by training only by collecting a small amount of actual scene images; in addition, the method adopts the marking tool which is developed independently, so that the data set marking work can be completed rapidly and the requirement of a subsequent elliptic parameter regression model on the data set can be met; the data set expansion method has high expansion speed, the effect of the synthesized image is close to that of a real image, and the method is suitable for industrial environments with continuously changing conditions and can be widely applied to more specific occasions;

the whole flow of the pin image generation method realized by the invention is shown in figure 23; generating an image by using the model under each resolution, wherein an example picture is shown in fig. 24;

3) Defect image generation and labeling

The final objective of the invention is to expand the pin image dataset for the training of a subsequent chain spacing measurement model; the main purpose is to train a generating model by utilizing the image collected by the initial camera, generate a corresponding image and label; therefore, after the pin shaft image is synthesized through the generated model, the image is marked by a marking tool according to the original data set manufacturing project.

The chain distance measuring method comprises a non-overlapping view field multi-camera combined calibration method, a flow chart is shown in fig. 25, and the specific operation steps are as follows:

non-overlapping field of view multi-cameras are shown in fig. 28;

the specific operation steps are as follows:

1) Monocular calibration and optimization thereof

The target calibration plate is placed under each camera view field, the positions of the target calibration plate are transformed for multiple times to obtain multiple images, the target calibration plate is guaranteed to appear at each position under the camera view field, then two-dimensional coordinates of the target are obtained through a target recognition and decoding algorithm, the two-dimensional coordinates are corresponding to the three-dimensional coordinates, and then accurate values of internal parameters, external parameters and distortion coefficients of the camera are obtained through the monocular calibration and binding adjustment algorithm provided by the invention.

2) Multi-camera joint calibration and binding adjustment optimization under non-overlapping view field

On the basis of monocular calibration, a target calibration plate is placed in the middle of an adjacent camera, and multiple images are acquired by moving the position of the target calibration plate for multiple times, so that the two cameras can recognize targets as much as possible, and the targets are uniformly distributed in the two camera markets. The precise values of the conversion matrix between all adjacent cameras are calculated by utilizing the multi-camera joint calibration algorithm and the optimization algorithm thereof.

Examples:

an 11 x 9 target calibration plate was selected, in which no duplicate targets were found, as shown in fig. 26 and 27. Monocular calibration and optimization treatment are carried out by using the target, so that accurate parameters of camera internal parameters, external parameters and distortion coefficients are obtained, and the final calibration result is shown in table 1.

TABLE 1 Camera 1 calibration and optimization result analysis

	Before optimization	After optimization
			f _x	3792.56	3794.55
f _y	3792.56	3794.55
			u ₀	1279.53	1280.74
v ₀	963.56	960.32
			k ₁	-0.0045	-0.0024
k ₂	0.004	0.0234
			p ₁	-0.00361	-0.00234
p ₂	0.0	-0.005
			k ₃	0.27	0.15
System reprojection error	0.241	0.0067

Examples:

based on the experimental results, multi-camera combined calibration and optimization experiments are carried out, the same target calibration plate is selected, the accurate result of the conversion relation between cameras is obtained, and the final partial calibration results are shown in table 2.

TABLE 2 Joint calibration between cameras 12 under non-overlapping fields of view and optimization result analysis thereof

	Before optimization	After optimization
			Rotation vector	[0.00698，-0.00165，0.0936]	[0.0071，-0.0015，-0.0013]
Translation vector	[-664.33，-1.56，-1.67]	[-665.11，-0.34，-1.46]
			System reprojection error	0.15	0.04

The implementation effect is as follows:

and (3) optimizing internal parameters and external participation distortion coefficients of the camera through a binding adjustment algorithm so as to minimize the reprojection error.

2) And (3) the multi-camera combined calibration and the optimization algorithm design thereof, and the binding adjustment is utilized to optimize calibration parameters so as to minimize the sum of the re-projection errors of all cameras. The camera parameters are calibrated through the coding target calibration plate which is designed autonomously, so that the joint calibration of multiple cameras under the condition of no overlapping view fields can be realized. The method has the advantages of low algorithm complexity, strong robustness, high calibration precision, high calibration speed and the like.

Overall, the following is true: two sets of machine vision sensor systems are erected at the positions of about 0.6 m on two sides of a chain, each set of system comprises a group of binocular vision systems formed by two industrial cameras, four cameras jointly form a four-eye camera vision system under a non-overlapping visual field, and simultaneously, lasers, light sources and the like are used for providing additional three-dimensional information or providing peripheral peripherals of a good image acquisition environment, so that the calculation of the three-dimensional coordinates of the center of a chain shaft is realized;

The foregoing is merely a preferred embodiment of the present invention, and it should be noted that modifications and substitutions can be made by those skilled in the art without departing from the technical principles of the present invention, and these modifications and substitutions should also be considered as being within the scope of the present invention.

Claims

1. The chain distance measuring system is characterized in that at least two sets of machine vision sensor systems are erected on two sides of a chain distance to be measured, each machine vision sensor system comprises a binocular vision system formed by two cameras, and each binocular vision system comprises a calibration unit and a measuring unit;

2. The chain pitch measurement system of claim 1, wherein the four-vision system identifies, tracks and locates a chain axle; when the tested chain is just in the central region of the image, the center of the chain shaft is extracted, and the elliptic regression network of the mask-CNN is directly fitted to the outline of the ellipse.

3. The chain pitch measurement system of claim 1, wherein a high-precision target is provided in combination for automatically detecting the system precision of the four-vision system; the calibration unit is operated for the equipment every 2 or 3 months; the four cameras of the two binocular systems form a four-vision system; the laser and the light source are arranged in a matching way, so that the calculation of the three-dimensional coordinate of the center of the chain shaft is realized; setting air blowing equipment and installing an industrial hairbrush module for blowing and cleaning a chain shaft; a tripod is arranged for supporting and fixing the camera and the light source; the height of the tripod is adjustable.

4. A chain pitch measurement method, applying the chain pitch measurement system according to any one of claims 1 to 3, characterized by comprising the steps of: a calibration step and a measurement step;

5. The chain distance measuring method as set forth in claim 4, comprising a vision system calibration method without overlapping fields of view, for vision system calibration in which the target information obtained by two vision systems is not in the same coordinate system, wherein all cameras are calibrated jointly, and all cameras together form a multi-camera vision system to obtain a reference camera coordinate system of the multi-camera system;

the method comprises the following steps:

s2, monocular calibration, wherein the calibration algorithm in S2 comprises the following steps: calibrating internal parameters of the camera and solving distortion coefficients to finish monocular calibration; the monocular calibration in S2 includes coordinate conversion: performing camera internal reference solving by using a Zhang Zhengyou internal reference calibration method to obtain a radial distortion coefficient and a tangential distortion coefficient of the camera, and correcting projection coordinates by using the distortion parameters;

6. The chain pitch measurement method according to claim 4, comprising a camera exposure time adaptive control method of line structured light image features, characterized by comprising an adaptive multiple exposure method;

multiple exposure process recordThe following parameters were: a confidence threshold T representing acceptable streak image quality; initial exposure time t _i The reliability of the gray level minimum position is larger than the threshold value T; the exposure time adjusting step length is delta t; the proportion K meeting the credibility threshold is the judging standard for ending shooting;

the adaptive multi-exposure method comprises the following steps D1-D4:

the light bar credibility evaluation method comprises the steps of G1-G4:

g2, fitting a Gaussian curve based on a light bar center point;

7. The chain pitch measurement method according to claim 4, comprising an ellipse detection method of fast R-CNN, comprising the steps of:

8. The chain pitch measurement method according to claim 4, comprising a pin image data enhancement method, comprising the steps of:

S1, making and marking an original data set, wherein the obtained images of the chain pin shafts have different angles and orientations; the method comprises data cleaning, data labeling and labeling inspection;

s2, generating a model, constructing a network, optimizing training, and training a StyleGAN3 generator on a pin shaft image data set; adjusting the generator and regularization loss thereof, and configuring corresponding matched ProjectedGAN and class conditions;

s2, generating and labeling a defect image;

s3, introducing gradual growth;

the StyleGAN3 generator consists of a mapping network Gm and a synthesis network Gs;

gm maps a normally distributed potential code z to a pattern code w;

modulating the convolution kernel of Gs with the pattern code w to control the synthesis process;

the synthesis network Gs of StyleGAN3 starts with a spatial mapping defined by fourier features, inputs are convolved through N layers, non-linearities, and upsampling to generate an image, each non-linearities being wrapped by upsampling and downsampling operations;

the system comprises a low-pass filter, wherein the low-pass filter is used for balancing image quality, anti-aliasing and training speed;

ProjectedGAN expands the original resistance game between generator G and discriminator D by a set of characteristic Projectors { Pl }; the Projector maps the images generated by the real images x and G to the input space of the discriminator;

ProjectedGAN targets are:

where { Dl } is a set of independent discriminators that handle different feature Projectors; the Projector consists of a trained feature network F, cross-channel mixing and cross-scale mixing layers;

CCM and CSM are used to prohibit the discriminator from focusing on only a subset of its input feature space;

CCM mixes the characteristic of the cross-channel through the random 1x1 convolution, CSM mixes the characteristic of the cross-scale through the residual random 3x3 convolution block and bilinear upsampling;

the output of CSM is a feature pyramid composed of four feature graphs with different resolutions; four discriminators operate independently on these feature maps; each discriminator uses a simple convolution structure and spectral normalization; the depth of the discriminator depends on the resolution of its input, i.e. a spatially larger feature map corresponds to a deeper discriminator; regularization of the Projected GAN with spectral normalization only;

applying 8 independent discriminators on the multi-scale feature map; the image is provided to a classifier CLF for classifier guidance;

and S2, training StyleGAN-XL on the original pin shaft image data set, and continuously adjusting the initial resolution to generate a real image contour.

9. The chain pitch measurement method of claim 4, comprising a non-overlapping field-of-view multi-camera joint calibration method, comprising the steps of:

s2, calibrating parameters of adjacent cameras;

the camera imaging model is:

wherein f _x ，f _y ，u ₀ ，v ₀ Is a camera internal parameter, R, t is a camera external parameter, (X) _W ，Y _W ，Z _W ) Is the coordinate under the world coordinate system;

the camera perspective imaging model is as follows:

the rotation vectors r1 and r2 are in unit orthogonal relationship, that is:

r ₁ ^T r ₂ ＝0

r ₁ ^T r ₁ ＝r ₂ ^T r ₂ ＝1

The method can obtain:

r ₁ ＝K ^-1 H ₁

r ₂ ＝K ^-1 H ₂

substituting the camera imaging model can obtain:

calculating the accurate value of the camera parameter in a mode of minimizing the system re-projection error by using a binding adjustment optimization method; the formula of the reprojection error is:

the calculation formula of the binocular camera calibration is as follows:

the pose relation between cameras is improved as follows:

wherein u is _i 、v _i Representing the two-dimensional coordinates of the target point on the real image,representing two-dimensional coordinates of target points re-projected to a pixel coordinate system according to camera parameters, and n represents the sum of the numbers of all target points. />