CN116543376B

CN116543376B - Pointer type instrument reading method and system

Info

Publication number: CN116543376B
Application number: CN202310799369.2A
Authority: CN
Inventors: 邹浩立; 周熺; 吴建光; 黄汉生; 黄德华; 黄磊; 董朕; 李凯; 谢浩南; 梁炳; 王征迪
Original assignee: Zhaoqing Power Supply Bureau of Guangdong Power Grid Co Ltd
Current assignee: Zhaoqing Power Supply Bureau of Guangdong Power Grid Co Ltd
Priority date: 2023-07-03
Filing date: 2023-07-03
Publication date: 2023-12-29
Anticipated expiration: 2043-07-03
Also published as: CN116543376A

Abstract

The invention discloses a pointer type instrument reading method and a pointer type instrument reading system. And dividing the scale marks of the initial instrument dial image to generate an initial scale mark division image. And respectively performing perspective transformation calibration on the initial instrument dial image and the corresponding initial scale mark segmentation image to generate a target instrument dial image and a target scale mark segmentation image. And (5) carrying out auxiliary circle construction by adopting a target scale mark segmentation image, and determining the dip angle of the scale mark. And calculating the pointer inclination angle of the dial image of the target instrument by adopting a maximum connected domain labeling method and a contour refinement algorithm to generate the pointer inclination angle. And determining a meter reading data set corresponding to the pointer type meter image data set by adopting all scale line dip angles and all pointer dip angles. The inclination angles of the scale marks of the minimum value and the maximum value of the dial can be obtained efficiently through the auxiliary circles, and the robustness is high.

Description

Pointer type instrument reading method and system

Technical Field

The invention relates to the technical field of instruments, in particular to a pointer instrument reading method and system.

Background

The pointer instrument has the advantages of vibration resistance, high temperature resistance, magnetic interference resistance, long service life, high reliability and the like, so that a large number of substations are still installed in China. The inspection mode of the instrument equipment in the domestic transformer substation is mainly manual inspection, the observation and reading of the instrument adopts naked eye observation and a simple instrument shooting method, the instrument data information adopts a handwriting recording mode, the working content is monotonously repeated, and the labor intensity is high. In addition, the transformer substation is located in suburban open field, and under severe conditions such as extreme weather and equipment radiation, certain potential safety hazards exist for patrol personnel.

With the development of information technology, inspection robots are adopted to replace manual inspection, so that the automation level of the transformer substation is greatly improved. And for a large number of instrument images acquired by the inspection robot, reading identification is carried out by adopting an automatic pointer instrument reading identification method. The identification flow of the method is roughly divided into three steps of instrument target detection, dial scale division and pointer position identification.

The traditional instrument target detection method mainly comprises the steps of finding image corner points by using gradients of images, and then carrying out feature matching with instrument images of a template library according to corner point detection results. But the corner detection method is easily affected by image noise and environmental shielding factors. In the aspect of scale mark segmentation and pointer identification, a Hough transformation method is adopted for scale mark segmentation and pointer identification, however, the Hough transformation has strong dependence on parameters and poor flexibility, and when an instrument image with complex background is processed, a great number of interference items appear in the Hough transformation detection result.

Disclosure of Invention

The invention provides a pointer type instrument reading method and a pointer type instrument reading system, which solve the technical problems that the existing pointer type instrument reading method has strong dependence on parameters, has poor flexibility, is easy to be interfered by external environment, and causes low accuracy of reading results.

The invention provides a pointer type instrument reading method, which comprises the following steps:

acquiring a pointer type instrument image data set, and carrying out instrument area identification by adopting the pointer type instrument image data set to generate an initial instrument dial image;

dividing the scale marks of the initial instrument dial image to generate an initial scale mark division image;

performing perspective transformation calibration on the initial instrument dial image and the corresponding initial scale mark segmentation image respectively to generate a target instrument dial image and a target scale mark segmentation image;

performing auxiliary circle construction by adopting the target scale mark segmentation image, and determining a scale mark inclination angle;

calculating a pointer inclination angle of the dial image of the target instrument by adopting a maximum connected domain labeling method and a contour refinement algorithm to generate a pointer inclination angle;

and determining a meter reading data set corresponding to the pointer type meter image data set by adopting all the scale line dip angles and all the pointer dip angles.

Optionally, the step of using the pointer instrument image dataset to identify an instrument area and generating an initial instrument dial image includes:

constructing a model training set and a model testing set by adopting the pointer instrument image data set;

respectively adopting dial area positions and dial type labels corresponding to all images in the model training set to construct a data exchange file corresponding to the images;

performing model training on a preset instrument area identification model by adopting the model training set and all the data exchange files to generate an initial instrument area identification model;

performing model test on the initial instrument area identification model by adopting the model test set to generate a target instrument area identification model;

and carrying out target recognition on the pointer instrument image data set through the target instrument area recognition model to generate an initial instrument dial image.

Optionally, the step of dividing the initial meter dial image into scale marks and generating an initial scale mark divided image includes:

dividing the pointer instrument image data set into a semantic segmentation training set and a semantic segmentation test set according to a preset dividing proportion;

Masking the semantic segmentation training set to generate a label image set;

adopting a plurality of transformer encoder modules as encoders and a plurality of convolution modules as decoders to construct an initial semantic segmentation model;

performing model training on the initial semantic segmentation model by adopting the semantic segmentation training set and the label image set to generate an intermediate semantic segmentation model;

performing model test on the intermediate semantic segmentation model by adopting the semantic segmentation test set to generate a target semantic segmentation model;

respectively carrying out Gaussian filtering and median filtering on the initial instrument dial image to generate an instrument dial filtering image;

and dividing the scale marks of the instrument dial plate filtering image through the target semantic division model to generate an initial scale mark division image.

Optionally, the step of performing perspective transformation calibration on the initial meter dial image and the corresponding initial scale mark segmentation image to generate a target meter dial image and a target scale mark segmentation image includes:

substituting all pixel point coordinates of the initial scale mark segmentation image into a preset ellipse equation to solve, and generating a fitting ellipse center;

Performing perspective transformation calculation by adopting the fitted ellipse center and the initial instrument dial image to generate a perspective transformation matrix;

and calibrating the initial instrument dial image and the corresponding initial scale mark segmentation image by adopting the perspective transformation matrix to generate a target instrument dial image and a target scale mark segmentation image.

Optionally, the step of performing auxiliary circle construction by using the target scale mark segmentation image to determine a scale mark inclination angle includes:

taking the fitted ellipse center as a circle center, and constructing an auxiliary circle according to a first preset radius;

rotating the auxiliary circle according to a preset rotation starting point and a rotation standard to construct an auxiliary circle list;

respectively constructing line segments corresponding to the rotation points by respectively connecting the rotation points in the auxiliary circle list with the circle center;

constructing an auxiliary circle intersection list by adopting all line segments intersected with the target scale line segmentation image;

and calculating the inclination angle between the dial minimum value and the dial maximum value scale mark in the auxiliary circle intersection list, and generating the scale mark inclination angle.

Optionally, the step of calculating the pointer inclination angle of the target instrument dial image by adopting a maximum connected domain labeling method and a contour refinement algorithm to generate the pointer inclination angle includes:

Performing gray level conversion on the dial image of the target instrument, and performing binarization by adopting a maximum inter-class variance method to generate a binary image;

taking the fitted ellipse center as a circle center, and constructing a circular mask image according to a second preset radius;

performing parity multiplication on the binary image and the circular mask image to generate an inner dial binary image;

performing morphological open operation on the inner dial binary image to generate a connected domain coordinate list, and taking the largest connected domain in the connected domain coordinate list as a pointer contour image;

adopting a contour refinement algorithm to refine contours of the pointer contour images, and determining pointer frameworks;

respectively calculating Euclidean distances between each pixel point in the pointer skeleton and the fitted ellipse center, and taking the point corresponding to the maximum value of the Euclidean distances as the pixel point of the pointer fingertip;

and calculating the pointer inclination angle by adopting the pointer fingertip pixel points to generate the pointer inclination angle.

Optionally, the step of determining the meter reading data set corresponding to the pointer meter image data set by using all the scale line inclination angles and all the pointer inclination angles includes:

respectively calculating the difference value between the fitted ellipse center and the finger tip vector of the pointer corresponding to the pointer inclination angle to generate a first difference value;

Respectively calculating the difference value between the coordinate vectors of the minimum values of the scale marks corresponding to the fitting ellipse center and the scale mark inclination angles, and generating a second difference value;

respectively calculating the difference between the maximum coordinate vectors of the scale marks corresponding to the fitted ellipse center and the scale mark inclination angles, and generating a third difference;

calculating an angle between the first difference value and the second difference value to generate a first angle;

calculating an angle between the second difference value and the third difference value to generate a second angle;

calculating the ratio between the first angle and the second angle and multiplying the ratio by the corresponding instrument measuring range to generate pointer reading;

and constructing a meter reading data set corresponding to the pointer meter image data set by adopting all pointer readings.

The invention also provides a pointer type instrument reading system, which comprises:

the initial instrument dial image generation module is used for acquiring a pointer instrument image data set, and carrying out instrument area identification by adopting the pointer instrument image data set to generate an initial instrument dial image;

the initial scale mark segmentation image generation module is used for performing scale mark segmentation on the initial instrument dial image to generate an initial scale mark segmentation image;

The target instrument dial image and target scale mark segmentation image generation module is used for performing perspective transformation calibration on the initial instrument dial image and the corresponding initial scale mark segmentation image respectively to generate a target instrument dial image and a target scale mark segmentation image;

the scale mark inclination angle generation module is used for calculating the scale mark inclination angle by adopting the target scale mark segmentation image to generate a scale mark inclination angle;

the pointer inclination angle generation module is used for calculating the pointer inclination angle of the dial image of the target instrument by adopting a maximum connected domain labeling method and a contour refinement algorithm to generate a pointer inclination angle;

and the instrument reading data set determining module is used for determining an instrument reading data set corresponding to the pointer type instrument image data set by adopting all the scale line dip angles and all the pointer dip angles.

Optionally, the initial meter dial image generating module includes:

the model training set and model test set construction module is used for constructing a model training set and a model test set by adopting the pointer instrument image data set;

the data exchange file construction module is used for constructing a data exchange file corresponding to each image by adopting the dial area position and the dial type label corresponding to each image in the model training set;

The initial instrument area identification model generation module is used for carrying out model training on a preset instrument area identification model by adopting the model training set and all the data exchange files to generate an initial instrument area identification model;

the target instrument area identification model generation module is used for carrying out model test on the initial instrument area identification model by adopting the model test set to generate a target instrument area identification model;

and the initial instrument dial image generation sub-module is used for carrying out target identification on the pointer instrument image data set through the target instrument area identification model to generate an initial instrument dial image.

Optionally, the initial tick mark segmentation image generation module includes:

the semantic segmentation training set and semantic segmentation test set generation module is used for dividing the pointer instrument image data set into a semantic segmentation training set and a semantic segmentation test set according to a preset dividing proportion;

the label image set generating module is used for carrying out mask processing on the semantic segmentation training set to generate a label image set;

the initial semantic segmentation model construction module is used for constructing an initial semantic segmentation model by adopting a plurality of transformer encoder modules as encoders and a plurality of convolution modules as decoders;

The intermediate semantic segmentation model generation module is used for carrying out model training on the initial semantic segmentation model by adopting the semantic segmentation training set and the label image set to generate an intermediate semantic segmentation model;

the target semantic segmentation model generation module is used for carrying out model test on the intermediate semantic segmentation model by adopting the semantic segmentation test set to generate a target semantic segmentation model;

the instrument dial filter image generation module is used for respectively carrying out Gaussian filter and median filter processing on the initial instrument dial image to generate an instrument dial filter image;

the initial scale mark segmentation image generation sub-module is used for performing scale mark segmentation on the instrument dial plate filtering image through the target semantic segmentation model to generate an initial scale mark segmentation image.

From the above technical scheme, the invention has the following advantages:

according to the invention, the pointer type instrument image data set is acquired, and the pointer type instrument image data set is adopted to identify the instrument area, so that an initial instrument dial image is generated. And dividing the scale marks of the initial instrument dial image to generate an initial scale mark division image. And respectively performing perspective transformation calibration on the initial instrument dial image and the corresponding initial scale mark segmentation image to generate a target instrument dial image and a target scale mark segmentation image. And (5) carrying out auxiliary circle construction by adopting a target scale mark segmentation image, and determining the dip angle of the scale mark. And calculating the pointer inclination angle of the dial image of the target instrument by adopting a maximum connected domain labeling method and a contour refinement algorithm to generate the pointer inclination angle. And determining a meter reading data set corresponding to the pointer type meter image data set by adopting all scale line dip angles and all pointer dip angles. The method solves the technical problems that the existing pointer instrument reading method has strong dependence on parameters, poor flexibility and low accuracy of reading results due to the fact that the existing pointer instrument reading method is easily interfered by external environments. The inclination angles of the dial minimum and maximum scale marks can be obtained efficiently through the auxiliary circles, and the method can still obtain accurate results under the condition of low scale mark segmentation accuracy, and has strong robustness.

Drawings

In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention, and that other drawings can be obtained from these drawings without inventive faculty for a person skilled in the art.

FIG. 1 is a flowchart showing steps of a method for reading a pointer meter according to an embodiment of the present invention;

FIG. 2 is a flowchart showing steps of a method for reading a pointer meter according to a second embodiment of the present invention;

fig. 3 is a block diagram of a pointer meter reading system according to a third embodiment of the present invention.

Detailed Description

The embodiment of the invention provides a pointer instrument reading method and a pointer instrument reading system, which are used for solving the technical problems that the existing pointer instrument reading method has strong dependence on parameters, has poor flexibility, is easy to be interfered by external environment and causes low accuracy of reading results.

In order to make the objects, features and advantages of the present invention more comprehensible, the technical solutions in the embodiments of the present invention are described in detail below with reference to the accompanying drawings, and it is apparent that the embodiments described below are only some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Referring to fig. 1, fig. 1 is a flowchart illustrating a pointer meter reading method according to an embodiment of the invention.

The first embodiment of the invention provides a pointer type instrument reading method, which comprises the following steps:

and 101, acquiring a pointer type instrument image data set, and carrying out instrument area identification by adopting the pointer type instrument image data set to generate an initial instrument dial image.

In the embodiment of the invention, after the pointer instrument image dataset is obtained, a model training set and a model testing set are constructed by adopting the pointer instrument image dataset. And respectively adopting dial area positions and dial type labels corresponding to all images in the model training set to construct a data exchange file corresponding to the images. And carrying out model training on the preset instrument area identification model by adopting the model training set and all data exchange files to generate an initial instrument area identification model. And performing model test on the initial instrument area identification model by using a model test set to generate a target instrument area identification model. And carrying out target recognition on the pointer instrument image dataset through the target instrument area recognition model to generate an initial instrument dial image.

And 102, dividing the scale marks of the initial instrument dial image to generate an initial scale mark divided image.

In the embodiment of the invention, firstly, a pointer instrument image dataset is divided into a semantic segmentation training set and a semantic segmentation test set according to a preset division ratio. And then masking the semantic segmentation training set to generate a label image set. By using a plurality of transformer encoder modules as encoders and a plurality of convolution modules as decoders, an initial semantic segmentation model is constructed. And carrying out model training on the initial semantic segmentation model by adopting a semantic segmentation training set and a label image set to generate an intermediate semantic segmentation model. And performing model test on the intermediate semantic segmentation model by adopting a semantic segmentation test set to generate a target semantic segmentation model. And carrying out Gaussian filtering and median filtering on the initial instrument dial image respectively to generate an instrument dial filtering image. And dividing the scale marks by the instrument dial filter image of the target semantic division model to generate an initial scale mark division image.

And 103, respectively performing perspective transformation calibration on the initial instrument dial image and the corresponding initial scale mark segmentation image to generate a target instrument dial image and a target scale mark segmentation image.

In the embodiment of the invention, all pixel point coordinates of the initial scale line segmented image are substituted into a preset ellipse equation to be solved, and a fitting ellipse center is generated. And performing perspective transformation calculation by adopting the fitted ellipse center and the initial instrument dial image to generate a perspective transformation matrix. And calibrating the initial instrument dial image and the corresponding initial scale mark segmentation image by adopting a perspective transformation matrix to generate a target instrument dial image and a target scale mark segmentation image.

And 104, carrying out auxiliary circle construction by adopting a target scale mark segmentation image, and determining the scale mark inclination angle.

In the embodiment of the invention, the fitted ellipse center is taken as the circle center, and an auxiliary circle is constructed according to a first preset radius. And rotating the auxiliary circle according to a preset rotation starting point and a rotation standard to construct an auxiliary circle list. And respectively constructing line segments corresponding to the rotation points by respectively combining the rotation points in the auxiliary circle list with the circle center. And constructing an auxiliary circle intersection list by adopting all line segments intersected with the target scale line segmentation image. And calculating the inclination angle between the dial minimum value and the dial maximum value scale mark in the auxiliary circle intersection list, and generating the scale mark inclination angle.

And 105, calculating the pointer inclination angle of the dial image of the target instrument by adopting a maximum connected domain labeling method and a contour refinement algorithm, and generating the pointer inclination angle.

In the embodiment of the invention, the target instrument dial image is subjected to gray level conversion and binarization by adopting a maximum inter-class variance method, so as to generate a binary image. And taking the fitted ellipse center as a circle center, and constructing a circular mask image according to a second preset radius. And performing parity multiplication on the binary image and the circular mask image to generate an inner dial binary image. And performing morphological open operation on the inner dial binary image to generate a connected domain coordinate list, and taking the largest connected domain in the connected domain coordinate list as a pointer contour image. And adopting a contour refinement algorithm to refine the contour of the pointer contour image, and determining a pointer skeleton. And respectively calculating Euclidean distances between each pixel point in the pointer skeleton and the center of the fitting ellipse, and taking the point corresponding to the maximum value of the Euclidean distances as the pixel point of the pointer fingertip. And calculating the pointer inclination angle by adopting the finger tip pixel points of the pointer to generate the pointer inclination angle.

And 106, determining a meter reading data set corresponding to the pointer type meter image data set by adopting all scale mark dip angles and all pointer dip angles.

In the embodiment of the invention, the difference between the fitted ellipse center and the finger tip vector of the pointer corresponding to the pointer inclination angle is calculated respectively to generate a first difference. And respectively calculating the difference value between the coordinate vectors of the minimum values of the scale marks corresponding to the fitting ellipse center and the scale mark inclination angle, and generating a second difference value. And respectively calculating the difference value between the maximum coordinate vectors of the scale marks corresponding to the fitting ellipse center and the scale mark inclination angle, and generating a third difference value. An angle between the first difference and the second difference is calculated, generating a first angle. And calculating the angle between the second difference value and the third difference value to generate a second angle. And calculating the ratio between the first angle and the second angle and multiplying the ratio by the corresponding meter measuring range to generate pointer reading. And constructing a meter reading data set corresponding to the pointer type meter image data set by adopting all pointer readings.

In the embodiment of the invention, the pointer type instrument image data set is acquired, and instrument area identification is carried out by adopting the pointer type instrument image data set, so that an initial instrument dial image is generated. And dividing the scale marks of the initial instrument dial image to generate an initial scale mark division image. And respectively performing perspective transformation calibration on the initial instrument dial image and the corresponding initial scale mark segmentation image to generate a target instrument dial image and a target scale mark segmentation image. And (5) carrying out auxiliary circle construction by adopting a target scale mark segmentation image, and determining the dip angle of the scale mark. And calculating the pointer inclination angle of the dial image of the target instrument by adopting a maximum connected domain labeling method and a contour refinement algorithm to generate the pointer inclination angle. And determining a meter reading data set corresponding to the pointer type meter image data set by adopting all scale line dip angles and all pointer dip angles. The method solves the technical problems that the existing pointer instrument reading method has strong dependence on parameters, poor flexibility and low accuracy of reading results due to the fact that the existing pointer instrument reading method is easily interfered by external environments. The inclination angles of the dial minimum and maximum scale marks can be obtained efficiently through the auxiliary circles, and the method can still obtain accurate results under the condition of low scale mark segmentation accuracy, and has strong robustness.

Referring to fig. 2, fig. 2 is a flowchart illustrating a pointer meter reading method according to a second embodiment of the invention.

Another method for reading a pointer instrument provided in the second embodiment of the present invention includes:

step 201, acquiring a pointer type instrument image data set, and adopting the pointer type instrument image data set to identify an instrument area so as to generate an initial instrument dial image.

Further, step 201 may include the following sub-steps S11-S15:

s11, constructing a model training set and a model testing set by adopting a pointer instrument image data set.

S12, respectively adopting dial area positions and dial type labels corresponding to all images in the model training set to construct a data exchange file corresponding to the images.

And S13, performing model training on the preset instrument area identification model by using the model training set and all data exchange files to generate an initial instrument area identification model.

S14, performing model test on the initial instrument area identification model by using a model test set to generate a target instrument area identification model.

And S15, carrying out target recognition on the pointer instrument image dataset through a target instrument area recognition model, and generating an initial instrument dial image.

In the embodiment of the invention, the preset instrument area identification model refers to a YOLOV5 model. The YOLOV5 data set is divided into a YOLOV5 training set and a YOLOV5 testing set according to a proportion, namely, the pointer instrument image data set is divided according to the dividing proportion set by actual needs, so that a model training set and a model testing set are constructed and obtained. And (3) making a JSON file for each picture of the Yolov5 training set to construct a data exchange file, wherein the JSON file records the position of a dial area and dial type labels in each image of the training set, namely respectively adopting the dial area position and dial type labels corresponding to each image in the model training set to construct the data exchange file corresponding to the image.

And carrying out model training on the preset instrument area identification model by adopting a model training set and all data exchange files, and adjusting parameters of the preset instrument area identification model to generate an initial instrument area identification model, wherein the parameters comprise a learning rate and training round numbers. And performing model test on the initial instrument area identification model by using a model test set, and taking the initial instrument area identification model meeting test standards as a target instrument area identification model. Target recognition is carried out on each image in the pointer type instrument image data set through the target instrument area recognition model, namely, original input instrument images are subjected toCutting to obtain corresponding meter dial image +.>。

And 202, dividing the scale marks of the initial instrument dial image to generate an initial scale mark divided image.

Further, step 202 may comprise the following sub-steps S21-S27:

s21, dividing the pointer instrument image data set into a semantic segmentation training set and a semantic segmentation test set according to a preset dividing proportion.

S22, carrying out mask processing on the semantic segmentation training set to generate a label image set.

S23, adopting a plurality of transformer encoder modules as encoders and a plurality of convolution modules as decoders to construct an initial semantic segmentation model.

S24, performing model training on the initial semantic segmentation model by adopting the semantic segmentation training set and the label image set to generate an intermediate semantic segmentation model.

S25, performing model test on the intermediate semantic segmentation model by using the semantic segmentation test set to generate a target semantic segmentation model.

S26, performing Gaussian filtering and median filtering on the initial instrument dial image respectively to generate an instrument dial filtering image.

And S27, dividing the scale marks of the instrument dial plate filtering image through the target semantic division model, and generating an initial scale mark division image.

The preset division ratio is the ratio of generating a semantic segmentation training set and a semantic segmentation test set based on the dial image data set of the division instrument set actually required, and the division ratio of the semantic segmentation training set and the semantic segmentation test set is generally set to be 8:2.

In the embodiment of the invention, a meter dial image dataset is adopted to manufacture a semantic segmentation model dataset based on ViT, and the meter dial image dataset is divided into a ViT training set and a ViT testing set, namely a semantic segmentation training set and a semantic segmentation testing set according to the proportion of 8:2. Making a mask image for each picture in the semantic segmentation training set, wherein the mask image reserves scale mark pixels in a dial area, and the label image is required to be preprocessed before model training: the background pixel type label is set to 0, and the target (tick mark) pixel type label is set to 1 in unity. And constructing a label image set by adopting all label images obtained through mask processing and preprocessing.

The semantic segmentation model based on ViT is built, namely an initial semantic segmentation model is built, and the initial semantic segmentation model comprises an encoder and a decoder. The encoder is a Vision Transformer model consisting of 12 Transformer encoder block (transformer encoder modules) and the decoder includes a plurality of convolution modules.

Let the label image input into the initial semantic segmentation model beFirst, the tag image is divided into fixed-size patches (image patches) of +.>Each label image will generate +>A patch, then the image small blocks are arranged into a sequence according to the line expansion, namely the sequence length of the input initial semantic segmentation model isEach patch is then mapped into a one-dimensional vector (token) and the tensor of the final input initial semantic segmentation model is +.>。

ViT-based segmentation model, namely initial semantic segmentation model, requires addition of position codes) Position deviation code->And input->An addition operation is performed. The position code is a table, the table has N rows, the size of N is the same as the sequence length of the input model, each row represents a vector, the dimension of the vector is the same as the dimension of the input sequence, and the formula is as follows:

；

wherein pos is the position number of the image patch in the sequence,is directed to an even dimension of the vector dimension, +. >Is an odd number of dimensions>Is the dimension of the model.

Each transformer encoder module in the encoder has the same structure, so that the input and the output of each transformer encoder module have the same dimension, and the transformer encoder module comprises a regularization Layer (LN), a multi-head attention layer (MSA) and a multi-layer perceptron layer (MLP), which respectively perform the functions of normalizing input data, performing global information interaction of all patches and enriching the dimension of each patch channel, and the calculation formula of the transformer encoder module is as follows:

；

wherein the method comprises the steps ofIs->Input of the individual transformer encoder modules, +.>Is->Intermediate variable of>Is->Output of (1), in particular->。

ViT decoder consists of one layerThe convolution layer and 4 blocks with the same structure are formed; />The convolution layer performs dimension reduction processing on the input features, so that the calculated amount is reduced; each block comprises two layers of convolution layers, and the convolution kernel size is +.>And an Upsampling layer;

ViT-based semantic segmentation model, namely loss function of initial semantic segmentation modelBy cross entropy loss->And a set similarity measure function->The formula is as follows:

；

wherein,，/>representing a sampleijProbability of being predicted as positive class, +.>Representing a sampleijThe positive class is 1, the negative class is 0, and N is the total number of samples; />，/>Representing pixel pointsiThe tag value of (1) in positive class and 0 in negative class,>representing pixel pointsiThe probability of being a positive class is predicted, N being the total number of pixel samples.

ViT training set and label image are input into ViT-based semantic segmentation model for training, and the function is lost after a plurality of iterationsAnd when the training is finished without reduction, training the initial semantic segmentation model by adopting the semantic segmentation training set and the label image set to generate an intermediate semantic segmentation model. And after training, testing the semantic segmentation model based on ViT by using ViT test set data, namely performing model test on the intermediate semantic segmentation model by using the semantic segmentation test set, so as to determine a target semantic segmentation model.

Because most pointer type meters are installed outdoors, various pollutants are attached to the surfaces of the meters, and therefore the quality of meter images shot by the inspection robot is uneven. In order to reduce the influence of low quality of the instrument image, gaussian filtering and median filtering processing can be performed on the initial instrument dial image to generate an instrument dial filtering image. Further, the scale mark segmentation is carried out on the instrument dial plate filtering image through the target semantic segmentation model, and an initial scale mark segmentation image is obtained 。

And 203, substituting all pixel point coordinates of the initial scale line segmentation image into a preset ellipse equation to solve, and generating a fitting ellipse center.

In an embodiment of the invention, the image is segmented at the initial tick markIn, the +.>The sitting mark of each pixel point is +.>Wherein->Indicate->Abscissa value of each pixel, ">Indicate->Ordinate values of the individual points. Establishing a preset elliptic equation: />. Let +.>Coordinates of individual pixels->Substituting the following equation into a preset elliptic equation, and solving the following equation:

；

finally by solving the above equationObtaining 5 parameters A, B, C, D and E of the ellipse equation, and further obtaining fitting ellipse center +.>。

And 204, performing perspective transformation calculation by adopting the fitted ellipse center and the initial instrument dial image to generate a perspective transformation matrix.

In the embodiment of the invention, the initial instrument dial image is calibrated by utilizing perspective transformation, and an imaginary plane where an actual dial is positioned is considered to fit an ellipse center(center of meter dial) as origin, establishing rectangular coordinate system in imaginary plane to +.>Drawing a circle for a central, preset radius R +.>Round->The four intersection points with the x and y axes are respectively: />、/>、、/>The method comprises the steps of carrying out a first treatment on the surface of the The four peaks of the long and short axes of the fitted ellipse are respectively marked as +. >、/>、、/>Projecting four vertexes of the long axis and the short axis of the fitted ellipse to the virtual plane by utilizing a perspective transformation formula, wherein the four vertexes are in one-to-one correspondence with four intersection pointsIn response, a perspective transformation matrix is obtained>。

The perspective transformation enables the projection of the initial meter dial image into three-dimensional space and the remapping to a new viewing plane. Let the homogeneous coordinates of a point on the original image plane beThe coordinates in the three-dimensional projection space are +.>In the new view plane the homogeneous coordinates are +.>The perspective transformation formula is:

；

wherein the transformation matrix，/>，。/>

And 205, calibrating the initial instrument dial image and the corresponding initial scale mark segmentation image by adopting a perspective transformation matrix to generate a target instrument dial image and a target scale mark segmentation image.

In the embodiment of the invention, the perspective transformation matrixBe applied to initial instrument dial plate image +.>And an initial tick mark segmentation image +.>Obtaining a target instrument dial image +.>And target tick mark segmentation image +.>。

And 206, constructing an auxiliary circle by adopting the target scale mark segmentation image, and determining the dip angle of the scale mark.

Further, the tick mark tilt angle comprises a tick mark minimum tilt angle and a tick mark maximum tilt angle, and step 206 may comprise the sub-steps S31-S35 of:

S31, taking the fitted ellipse center as a circle center, and constructing an auxiliary circle according to a first preset radius.

S32, rotating the auxiliary circle according to a preset rotation starting point and a rotation standard to construct an auxiliary circle list.

S33, respectively constructing line segments corresponding to the rotation points by the rotation points and the center of the circle in the auxiliary circle list.

S34, constructing an auxiliary circle intersection list by adopting all line segments intersected with the target scale line segmentation image.

S35, calculating the inclination angle between the dial minimum value and the dial maximum value scale mark in the auxiliary circle intersection list, and generating the scale mark inclination angle.

The first preset radius is the radius R of the circle A plus a preset constantI.e. the first preset radius is +.>。

The rotation standard is to rotate the preset rotation starting point anticlockwiseEvery rotation +.>Recording the rotation pointIs set in the coordinate value of (a).

In the embodiment of the present invention, the ellipse center is fitted in step 204For the center of the circle, construct auxiliary circle +.>The radius is +.>，/>Auxiliary circle->Equation>ToFor a preset rotation starting point, the rotation starting point is rotated counterclockwise +.>Every rotation +.>Recording the coordinate value of the rotation point, finally, obtaining the auxiliary round list +.>；

Get on auxiliary circle listFitting ellipse center point +. >Construction line segment->Use->Representing line segment->Dividing the image +.>Whether to intersect; if line segment->And->With intersection->Otherwise->According to the above rule, traverse +.>Constructing line segments to obtain auxiliary circle intersection list。

Based on the appearance characteristics of the scale marks of the instrument dial, namely, a larger gap exists between the minimum scale mark and the maximum scale mark. Thus, the auxiliary circle intersection listBecoming a head-to-tail connected torus, traversing from left to right looking for how many consecutive 0's each element exists right from its next bit. Finding the point on the minimum scale line according to the index subscript of the maximum valueI.e. dial minimum, auxiliary circle intersection list +.>Reverse order, the same principle can be obtained, points on the maximum scale lineI.e. the maximum graduation mark. And finally, generating the inclination angle of the scale marks by calculating the inclination angle between the minimum value and the maximum value of the dial in the auxiliary circle intersection list.

And 207, calculating the pointer inclination angle of the dial image of the target instrument by adopting a maximum connected domain labeling method and a contour refinement algorithm, and generating the pointer inclination angle.

Further, step 207 may include the following substeps S41-S47:

s41, performing gray level conversion on the dial image of the target instrument, and performing binarization by using a maximum inter-class variance method to generate a binary image.

S42, taking the fitted ellipse center as a circle center, and constructing a circular mask image according to a second preset radius.

S43, performing parity multiplication on the binary image and the circular mask image to generate an inner dial binary image.

S44, performing morphological open operation on the inner dial binary image to generate a connected domain coordinate list, and taking the largest connected domain in the connected domain coordinate list as a pointer contour image.

S45, adopting a contour refinement algorithm to refine contours of the contour images of the pointers, and determining the pointer skeleton.

S46, respectively calculating Euclidean distances between each pixel point in the pointer skeleton and the fitted ellipse center, and taking the point corresponding to the maximum value of the Euclidean distances as the pixel point of the pointer fingertip.

S47, calculating the pointer inclination angle by adopting the pixel points of the finger tips of the pointers, and generating the pointer inclination angle.

Second preset radius the radius R of the circle a minus a preset constantI.e. the second preset radius is +.>。

In the embodiment of the invention, the dial image of the target instrument is displayedConverting into gray level diagram, and binarizing by OTSU method, i.e. maximum inter-class variance method, to obtain binary diagram +.>. Fitting the ellipse center in step 204 +.>Is round center, is provided with circle->The radius is +.>Generating a circular mask image +. >I.e. the pixel value 1 in the circle and the pixel value 0 outside the circle. The binary diagram->And circular mask image->Co-located multiplication to obtain an inner dial binary pattern comprising only the inner dial +.>。

In an ideal case, the inner dial binary imageOnly the pointer profile pixels are included, but in practice some small interference points may remain, such as the pointer profile sticking to the numbers or graduation marks in the dial. In order to reduce the interference of the background of the inner dial, morphological processing can be performed on the binary image of the inner dial, the residual discrete small dot blocks are removed, and the pointer contour is separated from the incoherent pixel points. After morphological open operation, the remaining points are substantially within the pointer profile. The obtained inner dial binary diagram +.>Performing morphological opening operation, namely marking connected domain on the binary icon of the inner dial plate subjected to the morphological opening operation to obtain a connected domain coordinate list, finding out the longest item in the connected domain coordinate list, namely the largest connected domain, and taking the largest connected domain as a pointer contour image>。

By European distance formulaRespectively calculating all pixel points on the pointer skeleton and fitting ellipse centers in the step 204>The point corresponding to the maximum value is obtained and recorded as a pointer fingertip pixel point, and pointer inclination angle calculation is carried out by adopting the pointer fingertip pixel point and a corresponding instrument origin point, so as to generate a pointer inclination angle.

And step 208, determining a meter reading data set corresponding to the pointer type meter image data set by adopting all scale mark dip angles and all pointer dip angles.

Further, step 208 may include the following substeps S51-S57:

s51, respectively calculating the difference value between the fitting ellipse center and the finger tip vector of the pointer corresponding to the pointer inclination angle, and generating a first difference value.

S52, respectively calculating the difference value between the coordinate vectors of the minimum values of the scale marks corresponding to the fitting ellipse center and the scale mark inclination angle, and generating a second difference value.

S53, respectively calculating the difference between the maximum coordinate vectors of the scale marks corresponding to the fitting ellipse center and the scale mark inclination angles, and generating a third difference.

S54, calculating an angle between the first difference value and the second difference value, and generating a first angle.

S55, calculating an angle between the second difference value and the third difference value, and generating a second angle.

S56, calculating the ratio between the first angle and the second angle and multiplying the ratio by the corresponding instrument measuring range to generate pointer reading.

S57, adopting all pointer readings to construct a meter reading data set corresponding to the pointer meter image data set.

In the embodiment of the invention, based on the pointer inclination angle and the scale mark inclination angle of the dial plate, a final result of pointer instrument reading identification is obtained by adopting an angle method. Namely, based on the minimum coordinate of the scale mark, the maximum coordinate of the scale mark, the fingertip coordinate of the pointer and the center coordinate of the combined ellipse, the method can obtain: the angle alpha between the fingertip vector of the fitted ellipse center-pointer and the coordinate vector of the minimum value of the fitted ellipse center-scale line is calculated, namely the angle between the first difference value and the second difference value is calculated, and the first angle is generated. And (3) calculating the angle between the second difference value and the third difference value to generate a second angle by fitting the angle beta of the minimum coordinate vector of the ellipse center-scale line and the maximum coordinate vector of the ellipse center-scale line. And substituting the first angle and the second angle into a pointer reading formula of an angle method to obtain pointer reading, wherein the pointer reading formula is pointer reading theta= (alpha/beta) meter measuring range, namely calculating the ratio between the first angle and the second angle and multiplying the ratio by the corresponding meter measuring range to generate pointer reading. And finally, adopting all pointer readings to construct a meter reading data set corresponding to the pointer meter image data set.

In the embodiment of the invention, firstly, a YOLOV5 model is utilized to detect an instrument target area of an instrument image, the type of the instrument is obtained at the same time, then a Vision Transformer-based semantic segmentation model is used for dividing a scale line part of a dial plate area, then ellipse fitting is carried out according to scale line pixel points, and then a perspective transformation method is used for calibrating the dial plate image; dividing an image and an auxiliary circle by utilizing the perspective transformed scale marks to obtain the dip angles of the minimum and maximum scale marks of the dial; in the calibration dial plate image, an inner dial plate area is obtained, a pointer outline pixel point set is obtained by utilizing a maximum connected domain marking method, a pointer fingertip pixel point is obtained by judging the distance from a set element to a circle center, and finally, the inclination angle of a pointer is calculated, and the final identification result of the pointer type instrument is obtained by combining the inclination angle between the minimum and maximum scale marks of the dial plate and the type information of the instrument. YOLOV5 is a single-stage target detection model with low computer resource overhead and high real-time performance, and low omission ratio for small target (instrument area) detection. The adopted semantic segmentation model based on Vision Transformer reduces the complexity and cost of training model data annotation; the dial scale mark segmentation accuracy of different types of pointer instruments in a complex scene is improved; the model has strong generalization capability, and further expands the application range of the scheme of the invention. The inclination angles of the dial minimum and maximum scale marks can be obtained efficiently through the auxiliary circles, and the method can still obtain accurate results under the condition of low scale mark segmentation accuracy, and has strong robustness.

Referring to fig. 3, fig. 3 is a block diagram illustrating a pointer meter reading system according to a third embodiment of the present invention.

The third embodiment of the present invention provides a pointer meter reading system, including:

the initial meter dial image generating module 301 is configured to acquire a pointer meter image dataset, perform meter region identification by using the pointer meter image dataset, and generate an initial meter dial image.

The initial scale mark segmentation image generation module 302 is configured to segment the initial meter dial image into scale marks, and generate an initial scale mark segmentation image.

The target instrument dial image and target scale mark segmentation image generation module 303 is configured to perform perspective transformation calibration on the initial instrument dial image and the corresponding initial scale mark segmentation image, respectively, to generate a target instrument dial image and a target scale mark segmentation image.

The scale mark inclination angle generating module 304 is configured to perform scale mark inclination angle calculation by using the target scale mark segmentation image, and generate a scale mark inclination angle.

The pointer inclination angle generation module 305 is configured to perform pointer inclination angle calculation on the target instrument dial image by using a maximum connected domain labeling method and a contour refinement algorithm, so as to generate a pointer inclination angle.

The meter reading data set determining module 306 is configured to determine a meter reading data set corresponding to the pointer type meter image data set by using all the scale line tilt angles and all the pointer tilt angles.

Optionally, the initial meter dial image generation module 301 includes:

the model training set and model test set construction module is used for constructing a model training set and a model test set by adopting the pointer instrument image data set.

The data exchange file construction module is used for constructing the data exchange file corresponding to the image by adopting the dial area position and the dial type label corresponding to each image in the model training set.

The initial instrument area identification model generation module is used for carrying out model training on the preset instrument area identification model by adopting a model training set and all data exchange files to generate an initial instrument area identification model.

And the target instrument area identification model generation module is used for carrying out model test on the initial instrument area identification model by adopting the model test set to generate a target instrument area identification model.

Optionally, the initial tick mark segmentation image generation module 302 includes:

the system comprises a semantic segmentation training set and a semantic segmentation test set generation module, wherein the semantic segmentation training set and the semantic segmentation test set generation module are used for dividing the pointer instrument image data set into a semantic segmentation training set and a semantic segmentation test set according to a preset division proportion.

And the label image set generating module is used for carrying out mask processing on the semantic segmentation training set to generate a label image set.

The initial semantic segmentation model building module is used for building an initial semantic segmentation model by adopting a plurality of transformer encoder modules as encoders and a plurality of convolution modules as decoders.

The intermediate semantic segmentation model generation module is used for carrying out model training on the initial semantic segmentation model by adopting the semantic segmentation training set and the label image set to generate an intermediate semantic segmentation model.

The target semantic segmentation model generation module is used for carrying out model test on the intermediate semantic segmentation model by adopting the semantic segmentation test set to generate a target semantic segmentation model.

The instrument dial filter image generation module is used for respectively carrying out Gaussian filter and median filter processing on the initial instrument dial image to generate an instrument dial filter image.

Optionally, the target meter dial image and target tick mark segmentation image generation module 303 includes:

the fitted ellipse center generation module is used for substituting all pixel point coordinates of the initial scale line segmentation image into a preset ellipse equation to solve, and a fitted ellipse center is generated.

The perspective transformation matrix generation module is used for carrying out perspective transformation calculation by adopting the fitting ellipse center and the initial instrument dial plate image to generate a perspective transformation matrix.

The target instrument dial image and target scale mark segmentation image generation submodule is used for calibrating the initial instrument dial image and the corresponding initial scale mark segmentation image by adopting a perspective transformation matrix to generate a target instrument dial image and a target scale mark segmentation image.

Optionally, the tick mark dip angle generation module 304 includes:

the auxiliary circle construction module is used for constructing an auxiliary circle according to a first preset radius by taking the fitted ellipse center as a circle center.

The auxiliary circle list construction module is used for rotating the auxiliary circles according to a preset rotation starting point and a rotation standard to construct an auxiliary circle list.

And the line segment construction module is used for respectively constructing line segments corresponding to the rotation points by respectively combining the rotation points in the auxiliary circle list with the circle center.

And the auxiliary circle intersection list construction module is used for constructing an auxiliary circle intersection list by adopting all line segments intersected with the target scale line segmentation image.

And the scale mark inclination angle generation sub-module is used for calculating the inclination angle between the minimum value and the maximum value of the dial in the auxiliary circle intersection list and generating the scale mark inclination angle.

Optionally, the pointer inclination generating module 305 includes:

and the binary image generation module is used for carrying out gray level conversion on the dial image of the target instrument and carrying out binarization by adopting a maximum inter-class variance method to generate a binary image.

And the circular mask image construction module is used for constructing a circular mask image according to a second preset radius by taking the fitted ellipse center as a circle center.

And the inner dial binary image generation module is used for carrying out parity multiplication on the binary image and the circular mask image to generate an inner dial binary image.

The pointer contour image determining module is used for carrying out morphological open operation on the inner dial binary image, generating a connected domain coordinate list and taking the largest connected domain in the connected domain coordinate list as a pointer contour image.

The pointer skeleton determining module is used for carrying out contour refinement on the pointer contour image by adopting a contour refinement algorithm to determine the pointer skeleton.

The pointer fingertip pixel point determining module is used for respectively calculating Euclidean distances between each pixel point in the pointer skeleton and the fitted ellipse center, and taking the point corresponding to the maximum value of the Euclidean distances as the pointer fingertip pixel point.

The pointer inclination angle generation sub-module is used for calculating the pointer inclination angle by adopting the pixel points of the pointer fingertips to generate the pointer inclination angle.

Optionally, the meter reading dataset determination module 306 includes:

the first difference generating module is used for respectively calculating differences between the fitting ellipse center and pointer fingertip vectors corresponding to the pointer inclination angles to generate first differences.

And the second difference generating module is used for respectively calculating the difference between the coordinate vectors of the minimum values of the scale marks corresponding to the fitting ellipse center and the scale mark inclination angle to generate a second difference.

And the third difference generating module is used for respectively calculating the difference between the fitted ellipse center and the maximum coordinate vector of the scale mark corresponding to the scale mark inclination angle to generate a third difference.

The first angle generation module is used for calculating the angle between the first difference value and the second difference value and generating a first angle.

And the second angle generating module is used for calculating the angle between the second difference value and the third difference value and generating a second angle.

And the pointer reading generation module is used for calculating the ratio between the first angle and the second angle and multiplying the ratio by the corresponding instrument measuring range to generate pointer reading.

The meter reading data set determining sub-module is used for constructing a meter reading data set corresponding to the pointer type meter image data set by adopting all pointer readings.

The above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of pointer meter reading, comprising:

determining a meter reading data set corresponding to the pointer type meter image data set by adopting all the scale mark dip angles and all the pointer dip angles;

the step of dividing the scale marks of the initial instrument dial image to generate an initial scale mark divided image comprises the following steps:

masking the semantic segmentation training set to generate a label image set;

the loss function corresponding to the initial semantic segmentation model is as follows:

；

Wherein cross entropy loss，/>Representing a sampleijProbability of being predicted as positive class, +.>Representing a sampleijThe positive class is 1, the negative class is 0,Nis the total number of samples; aggregate similarity measure function->，/>Representing pixel pointsiThe tag value of (1) in positive class and 0 in negative class,>representing pixel pointsiThe probability of being predicted as a positive class,Nsample the total number of samples for the pixel;

performing model training on the initial semantic segmentation model by adopting the semantic segmentation training set and the label image set, and generating an intermediate semantic segmentation model when the loss function is not reduced any more;

2. The method of pointer meter reading of claim 1 wherein said step of using said pointer meter image dataset for meter area identification generates an initial meter dial image comprises:

3. The method of pointer meter reading of claim 1 wherein the step of generating a target meter dial image and a target tick mark segmentation image by perspective transformation calibration of the initial meter dial image and the corresponding initial tick mark segmentation image, respectively, comprises:

4. A pointer meter reading method according to claim 3, wherein said step of using said target tick mark segmentation image to assist in circle construction and determining the dip angle of the tick mark comprises:

5. The method for reading a pointer instrument according to claim 3, wherein the step of calculating the pointer inclination angle of the target instrument dial image by using a maximum connected domain labeling method and a contour refinement algorithm to generate the pointer inclination angle comprises the steps of:

6. A pointer meter reading method according to claim 3, wherein said step of determining a meter reading dataset corresponding to said pointer meter image dataset using all of said tick mark inclinations and all of said pointer inclinations comprises:

7. A pointer meter reading system, comprising:

the instrument reading data set determining module is used for determining an instrument reading data set corresponding to the pointer type instrument image data set by adopting all the scale line dip angles and all the pointer dip angles;

the initial tick mark segmentation image generation module comprises:

；

the intermediate semantic segmentation model generation module is used for carrying out model training on the initial semantic segmentation model by adopting the semantic segmentation training set and the label image set, and generating an intermediate semantic segmentation model when the loss function is not reduced any more;

8. The pointer meter reading system of claim 7 wherein said initial meter dial image generation module comprises: