CN115205858A - Pointer type instrument automatic reading method based on rotating target detection - Google Patents
Pointer type instrument automatic reading method based on rotating target detection Download PDFInfo
- Publication number
- CN115205858A CN115205858A CN202210807428.1A CN202210807428A CN115205858A CN 115205858 A CN115205858 A CN 115205858A CN 202210807428 A CN202210807428 A CN 202210807428A CN 115205858 A CN115205858 A CN 115205858A
- Authority
- CN
- China
- Prior art keywords
- pointer
- image
- dial
- area
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/14—Image acquisition
- G06V30/1444—Selective acquisition, locating or processing of specific regions, e.g. highlighted text, fiducial marks or predetermined fields
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/14—Image acquisition
- G06V30/148—Segmentation of character regions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/16—Image preprocessing
- G06V30/162—Quantising the image signal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/19—Recognition using electronic means
- G06V30/191—Design or setup of recognition systems or techniques; Extraction of features in feature space; Clustering techniques; Blind source separation
- G06V30/19173—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/19—Recognition using electronic means
- G06V30/191—Design or setup of recognition systems or techniques; Extraction of features in feature space; Clustering techniques; Blind source separation
- G06V30/1918—Fusion techniques, i.e. combining data from various sources, e.g. sensor fusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/02—Recognising information on displays, dials, clocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to the technical field of visual measurement, in particular to a pointer type instrument automatic reading method based on rotating target detection, which mainly comprises the following steps: extracting a dial area and carrying out perspective transformation; detecting and refining a pointer; scale value identification; and (5) reading calculation. The method utilizes Fast SCNN network to extract dial area and perform perspective correction, and provides high-quality dial image for subsequent identification work; construction based on R 3 The improved rotating target detection network of the Det acquires a pointer detection frame and has the characteristic of high precision; after the scales of the pointer framework and the dial are obtained, the angle is adoptedAnd (4) calculating the reading to realize automatic reading. The pointer type instrument automatic reading method based on the rotating target detection can accurately position the pointer area, corrode in the area in the direction to remove interference and fit the straight line where the pointer is located, greatly improves the pointer extraction precision, and particularly has a good detection effect on narrow pointers.
Description
Technical Field
The invention belongs to the field of research of visual measurement technology, and particularly relates to an automatic reading method of a pointer instrument based on rotary target detection.
Background
Aiming at the reading of the pointer instrument, the manual interpretation method is easily influenced by external factors and subjective factors of people, so that a reading result has larger error. In addition, in some heavy industrial sites which need frequent reading, such as chemical industry, metallurgy and the like, the workload of manual reading is huge and the content is single, so that time and labor are wasted, and the accuracy and rapidity of reading cannot be guaranteed. Meanwhile, some instruments may be in a high-temperature, high-voltage or high-radiation area, and manual reading brings a series of hidden dangers to the safety production. Therefore, the automatic reading identification of the pointer instrument has important practical significance.
The extraction of the meter pointer is one of the key steps affecting the reading accuracy. The method for extracting the pointer can be roughly divided into two methods, one method is a traditional image processing algorithm, namely, after interference is removed, a straight line where the pointer is located is determined by using methods such as straight line detection, template matching and the like, the adaptability of the method is often poor, and when background patterns in the dial plate are complex, the detection accuracy rate can be severely reduced due to excessive interference; the other is a deep learning technique, which mainly uses a semantic segmentation algorithm to obtain a pointer region, but when the pointer is narrow, a good segmentation effect cannot be obtained, and the pointer pixel is easily classified as a background.
Disclosure of Invention
Aiming at the defects of the prior art, the invention researches a pointer type instrument automatic reading method based on rotating target detection, utilizes the improved rotating target detection to accurately position a pointer area, carries out corrosion in the area with the direction to remove interference and fits a straight line where the pointer is positioned, and greatly improves the precision of pointer extraction.
In order to achieve the purpose, the technical scheme adopted by the invention is as follows.
A pointer instrument automatic reading method based on rotating target detection comprises the following steps:
the method comprises the following steps of (1) extracting a dial area and carrying out perspective correction to obtain a dial area correction image;
and (2) detecting and refining the pointer, wherein the specific steps are as follows:
step 2.1) construction and training based on R 3 The improved rotating target detection network of the Det obtains a pointer rotating bounding box through the rotating target detection network;
step 2.2) a mask of the pointer area is constructed according to the pointer rotation bounding box, the mask is superposed on the dial area correction image, and threshold segmentation is carried out to obtain a binary image only containing the pointer area;
step 2.3) carrying out corrosion with direction and connected domain marking on the binary image only containing the pointer region obtained in the step 2.2), and carrying out least square straight line fitting on the largest connected domain to obtain a pointer skeleton;
step (3), scale value identification;
and (4) reading calculation.
Further, the specific steps of the step (1) are as follows:
step 1.1), inputting an original image into a trained Fast SCNN network to generate a dial area mask image, wherein the dial area mask image is a binary image, white in the mask image is a dial area predicted by the network, and black is a background;
step 1.2), carrying out ellipse detection on the dial area mask image to obtain a fitting ellipse of the dial outline, and respectively recording the central coordinate, the major semi-axis, the minor semi-axis and the inclination angle of the dial ellipse as (x) c ,y c )、a、b、θ；
Step 1.3), using the center as (x) c ,y c ) Intercepting a dial area image by a square frame with the side length of 2 a;
step 1.4), the central coordinates (x) of the ellipse of the dial are used c ,y c ) The long half shaft a, the short half shaft B and the inclination angle theta, and calculating coordinates of four vertexes A, B, C, D, A ', B', C 'and D' of the dial ellipse before and after perspective transformation:
in the formula, points A and B are the vertexes of the major semi-axis of the ellipse of the dial plate, and points C and D are the vertexes of the minor semi-axis of the ellipse of the dial plate;
step 1.5), obtaining a perspective transformation matrix M by using four groups of vertex coordinates before and after transformation in the previous step, wherein the perspective transformation is a process of converting a three-dimensional object in a space coordinate system into a two-dimensional image for representation, and the complete form is as follows:
and u and v are coordinates of a certain point of the original image, the right third-order matrix is a perspective transformation matrix, and the coordinates of the corresponding image obtained after perspective transformation are x and y. The above equation is transformed into an equation form, namely:
substituting the coordinates of the four groups of vertexes into the equation to obtain eight equation sets, obtaining eight parameters by the simultaneous equation sets, and obtaining a perspective transformation matrix M, wherein a 33 And =1. And finally, performing dot multiplication on the dial area image and the perspective transformation matrix M to obtain a dial area correction image.
Further, the step 2.1) comprises the following specific steps:
step 2.1.1), performing Batch Norm normalization processing on each convolution layer of the MobileNet v2 model; respectively leading out output characteristic diagrams of 3 rd, 4 th and 6 th bottleeck structures of the MobileNetv2 model as the input of an FPN structure in an original network; m to be definedThe obileNetv2 model is instantiated as R 3 The Det model backbone network completes the replacement of the backbone network, and realizes the lightweight model; improving an inverted residual block of the MobileNet v2 network, adding an efficient attention module ECA-net after a Depth-wise convolution module, multiplying a weight vector output by the ECA-net and a Depth-wise convolution result in a channel dimension, and finally inputting the result to a subsequent Point-wise convolution module; constructing a rotary target detection network comprising an improved MobileNet v2 backbone network, an FPN characteristic pyramid structure, a characteristic refining module and a rotary target detector; the improved MobileNetv2 backbone network is used for replacing the original ResNet backbone network and is used for extracting the image characteristics of the dial area; the method comprises the steps that the features extracted by a backbone network are acquired layer by the FPN feature pyramid structure, the features of different scales are fused, and the fused features are used as the input of a feature refining module; the characteristic refining module recodes the position information of the pointer bounding box and reconstructs the whole characteristic diagram to realize characteristic alignment, so that the detected pointer bounding box is more accurate; the rotating target detector achieves detection of the bounding box with the direction and classification of the target.
Step 2.1.2), cutting out a dial plate image from the collected instrument image, removing a background, and expanding the data set in a zooming, rotating and overturning manner to obtain a second data set; labeling the second data set by adopting an image labeling tool Labelme to obtain a position label file of the pointer area, dividing the dial plate image and the corresponding position label file into a training set and a test set according to 4:1, inputting the training set into the rotating target detection network constructed in the step 2.1.1) for training, and testing the network by using the test set; inputting the dial plate area correction image obtained in the step 1.3) into a trained network to obtain a pointer rotation bounding box.
Further, the specific steps of step 2.2) are as follows:
step 2.2.1), calculating the rotation angle theta by using the vertex coordinates of the pointer rotation bounding box r The coordinates of the four vertexes are sequentially marked as (x) 1 ,y 1 )、(x 2 ,y 2 )、(x 3 ,y 3 )、(x 4 ,y 4 ) Calculating the angle of the longest edge, and setting the vertex (x) 1 ,y 1 )、(x 2 ,y 2 ) The longest side, the rotation angle theta r The calculation formula of (2) is as follows:
through a rotation angle theta r Obtaining a rotation matrix R 1 Then using the rotation matrix R 1 Rotating and converting the dial area correction image, rotating the pointer to a horizontal position, intercepting the pointer area image by using an image slicing method, and then performing Otsu threshold segmentation on the pointer area image to obtain an optimal segmentation threshold T;
step 2.2.2), a blank image with the size consistent with the dial area correction image and all pixel values of 0 is created, a pointer area is drawn on the blank image by using the vertex coordinates of the pointer rotation bounding box, and the filled pixel value is 255, so that a mask image of the pointer area can be obtained;
step 2.2.3), the mask image is superposed on the dial area correction image, and the threshold value T is utilized to carry out threshold value segmentation on the new image, so as to obtain a binary image only containing a pointer area;
further, the specific steps of step 2.3) are as follows:
step 2.3.1), respectively constructing structure elements in the directions of 0 degrees, 25 degrees, 45 degrees, 75 degrees, 90 degrees, 115 degrees, 135 degrees and 160 degrees, and rotating the structure elements by a rotation angle theta of a boundary box with the pointer r Taking the structure element closest to the angle as a structure element for corrosion operation, and performing directional corrosion on the binary image which only contains the pointer region and is obtained in the step 2.2);
step 2.3.2), marking the connected domain of the binary image subjected to the strip-direction corrosion, obtaining a coordinate list label _ dit of the connected domain, finding out the longest item in the label _ dit, namely the point set contained in the maximum connected domain, performing least square straight line fitting on all points in the point set to obtain a pointer skeleton, and marking the coordinates of the endpoint of the pointer as the coordinates of the pointer
Further, the step (3) comprises the following specific steps: firstly, inputting a dial plate area correction image into a trained EAST text detection network to obtain coordinates of scale value text bounding boxes on the dial plate, then sequentially intercepting scale value text images according to the coordinates and inputting the scale value text images into a CRNN network for recognition to obtain each scale value.
Further, the step (4) comprises the following specific steps: firstly, carrying out circle fitting on the central point of a text boundary box of the dial scale value, and recording the center of a fitting circle as (x) c ,y c ) The radius is recorded as R, and the pointer endpoint obtained in the step 2.3) is calculated
Taking the end point with small distance as the tip of the pointer and recording as P; then, sequentially calculating the distance from the central point of each scale value text bounding box to the tip P, finding two points closest to P, and recording the distance as d 1 、d 2 The corresponding scale value is recorded as k 1 、k 2 (ii) a Then, calculating the distance from the P to the center of the fitting circle, and recording the distance as d p By using the cosine theorem, the scale k can be respectively calculated 1 、k 2 Angle theta with the pointer 1 、θ 2 ：
Finally, calculating the reading k according to the angle proportional relation p ：
The invention has the beneficial effects that:
the invention provides a pointer instrument automatic reading method based on rotating target detection, which firstly utilizes Fast SCNN network to extract dial plate area and carries out perspective correction to provide high for subsequent identification workA quality dial image; reconstruction based on R 3 The improved rotating target detection network of the Det obtains the pointer rotating bounding box and has the characteristic of high precision. The method has the advantages that the rotating area where the pointer is located is accurately positioned, interference of patterns in other areas on the dial plate is eliminated, corrosion in the belt direction, connected domain marking and least square straight line fitting are conducted in the pointer area, interference is further eliminated, integrity of the pointer is guaranteed, accuracy of pointer extraction is greatly improved, particularly, the method has a good detection effect on narrow pointers, and automatic reading is achieved. Experiments show that the average reading accuracy of the invention can reach more than 98%.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a working flow chart of the automatic reading method of the pointer instrument of the invention.
Fig. 2 is a diagram showing the effect of image interception and perspective correction of the dial area according to the embodiment of the present invention.
FIG. 3 shows the invention based on R 3 Det's improved rotating object detection network architecture.
FIG. 4 is a diagram illustrating the effect of detecting pointer areas according to an embodiment of the present invention.
Fig. 5 is a graph of pointer effect extracted by the dial plate picture masking, threshold segmentation, directional etching and least square line fitting according to the embodiment of the invention.
Fig. 6 is a scale recognition effect diagram according to an embodiment of the present invention.
FIG. 7 is a schematic view of a reading calculation according to an embodiment of the present invention.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and thus the present invention is not limited to the specific embodiments disclosed below.
In the embodiment, the working process of the method for automatically reading the pointer instrument based on the detection of the rotating target is shown in fig. 1, and the method comprises the following steps:
step (1), dial plate area extraction and perspective correction, the specific steps are as follows:
step 1.1) making a data set, and training and testing the Fast SCNN network by using the data set
(a) Collecting instrument images containing different complex backgrounds and different measuring ranges, ensuring that partial images have the conditions of inclination and bright and shadow interference, and expanding a data set in a cutting, rotating and turning mode to obtain a first data set;
(b) Marking the instrument image in the first data set by adopting an image marking tool Labelme to obtain a corresponding json file, and then calling a Labelme _ json _ to _ dataset.exe program to sequentially extract a mask image of the dial plate from the json file, wherein a red area formed in the mask image is a polygonal area surrounded by marking points, namely a position label of the dial plate area;
(c) Dividing the instrument image in the first data set and the corresponding mask image into a training set and a testing set according to 4:1, inputting the training set into a Fast SCNN network for training, adjusting parameters of the network, wherein the parameters comprise learning rate, batch _ size and epoch, stopping training when loss does not decrease any more, and preventing overfitting; and testing the Fast SCNN network by using the test set.
Step 1.2) inputting an original image (namely an actual image acquired on site) into a trained Fast SCNN network to generate a dial area mask image, wherein white is a mask area, namely the predicted dial area of the network, and black is a background.
Step 1.3) perspective correction
(a) Carrying out ellipse detection on the mask image of the dial area to obtain the central coordinate (x) of the dial ellipse c ,y c ) A major semi-axis a, a minor semi-axis b and an inclination angle theta;
(b) Using the center as (x) c ,y c ) Intercepting an image of a dial area by a square frame with the side length of 2 a;
(c) Using the central coordinates (x) of the ellipse of the dial c ,y c ) The major semi-axis a, the minor semi-axis B and the inclination angle theta, and calculating coordinates of four vertexes A, B, C, D and A ', B', C 'and D' of the dial ellipse before and after perspective transformation:
in the formula, the point A and the point B are vertexes of a long semi-axis and a short semi-axis of an ellipse of the front dial plate subjected to perspective transformation, and the point C and the point D are vertexes of a short semi-axis and a short semi-axis of the ellipse of the front dial plate subjected to perspective transformation; the point A 'and the point B' are vertexes of a long semi-axis and a short semi-axis of the ellipse of the dial plate after perspective transformation, and the point C 'and the point D' are vertexes of a short semi-axis and a short semi-axis of the ellipse of the dial plate after perspective transformation;
(d) Obtaining a perspective transformation matrix M by using 4 groups of vertex coordinates before and after transformation in the last step, wherein the perspective transformation is a process of converting a three-dimensional object in a space coordinate system into a two-dimensional image for representation, and the complete form is as follows:
and u and v are coordinates of a certain point of the original image, the right third-order matrix is a perspective transformation matrix M, and the coordinates of the corresponding image obtained after perspective transformation are x and y. Transforming the above equation into an equation form, i.e.:
substituting 4 sets of point pairs (namely coordinates of four vertexes of the dial ellipse before and after perspective transformation) into the equation to obtain 8 equation sets, obtaining 8 parameters by simultaneous equation sets, and obtaining a perspective transformation matrix M, wherein a 33 And =1. Finally, the dial area image and the perspective transformation matrix M are subjected to dot multiplication to obtain dial area correctionAnd (4) an image. Fig. 2 shows the effect of partial dial area image interception and perspective correction.
And (2) detecting and refining the pointer, wherein the specific steps are as follows:
step 2.1) construction based on R 3 Improved rotating target detection network for Det
(a) Performing Batch Norm normalization processing on each convolution layer of the MobileNetv2 model; respectively leading out output characteristic diagrams of 3 rd, 4 th and 6 th bottleeck structures of the MobileNetv2 model as the input of an FPN structure in an original network; instantiating a well-defined MobileNetv2 model class as R 3 The Det model backbone network completes the replacement of the backbone network, and realizes the lightweight model;
(b) For the modification of the inverted residual block of the MobileNet v2 network, an attention module ECA-net is added after a Depth-wise convolution module, then a weight vector output by the ECA-net is multiplied by a Depth-wise convolution result in a channel dimension, and finally the result is input to a subsequent Point-wise convolution module.
(c) Improved R 3 The Det network structure is shown in fig. 3 and comprises a modified MobileNetv2 backbone network, an FPN feature pyramid structure, a feature refining module and a rotating object detector. The improved MobileNet v2 backbone network is used for replacing the original ResNet backbone network, so that the model has a simplified network structure and a faster reasoning speed and is used for extracting the image characteristics of the dial area; the FPN characteristic pyramid structure acquires characteristics extracted by the backbone network layer by layer, and fuses different scale characteristics, and the fused characteristics are used as input of a characteristic refining module to improve the detection capability of a multi-scale object; the feature refining module recodes the position information of the boundary box and reconstructs the whole feature map to realize feature alignment, so that the detected boundary box is more accurate; the rotating target detector achieves detection of the oriented bounding box and classification of the target.
Step 2.2) Making pairs of datasets based on R 3 Det's improved rotating target detection network for training and testing
(a) Cutting out a dial plate image from the collected instrument image, removing a background, and expanding a data set in a zooming, rotating and overturning mode to obtain a second data set;
(b) And labeling the second data set by adopting an image labeling tool Labelme, wherein each image corresponds to a labeling frame and a json file, and writing the coordinates of the labeling frame into the json file. The label box only contains the area from the pointer tip to the spindle edge. Then, sequentially extracting four vertex coordinates of the labeling frame from each json file
Respectively writing the images into txt files corresponding to the images to be used as training labels;
(c) Dividing the dial image and the corresponding txt label file into a training set and a testing set according to 4:1, inputting the training set into the improved rotating target detection network constructed in the step 2.1) for training, adjusting parameters of the network, wherein the parameters comprise learning rate, batch _ size and epoch, stopping training when loss does not decrease any more, and preventing overfitting; testing the network by using a test set; and finally, inputting the dial plate area correction image obtained in the step 1.3) into the network to obtain a pointer rotation boundary box. Fig. 4 shows the effect of the detection of the pointer area of the part of the dial image.
Step 2.3) obtaining a binary image only containing a pointer area
(a) Calculating a rotation angle theta by using the vertex coordinates of the pointer rotation bounding box obtained in the step 2.2) r The coordinates of the four vertexes are sequentially marked as (x) 1 ,y 1 )、(x 2 ,y 2 )、(x 3 ,y 3 )、(x 4 ,y 4 ) Calculating the angle of the longest edge, and setting the vertex (x) 1 ,y 1 )、(x 2 ,y 2 ) The longest side, the rotation angle theta r The calculation formula of (2) is as follows:
through a rotation angle theta r Obtaining a rotation matrix R 1 ：
Then, the coordinates of each point of the dial area correction image and the rotation matrix R are compared 1 Dot multiplication is carried out: [ x' i y′ i 1] T ＝R 1 *[x i y i 1] T In the formula (x) i ,y i ) The coordinates of each point of the image are corrected for the dial area and the pointer can be rotated to a horizontal position. Similarly, the four vertex coordinates of the pointer rotation bounding box are respectively matched with the rotation matrix R 1 Dot product to obtain rotated coordinates (x) 1 ',y 1 ')、(x 2 ',y 2 ')、(x 3 ',y 3 ')、(x 4 ',y 4 ') then use the method commonly used by OpenCv to cut out rectangular areas: img [ y) start :y end ,x start :x end ](img[y 3 ':y 1 ',x 3 ':x 1 ']Or img [ y 4 ':y 2 ',x 4 ':x 2 ']) Intercepting the dial area correction image after rotation transformation to obtain a horizontal pointer area image, and then performing Otsu threshold segmentation on the horizontal pointer area image to obtain an optimal segmentation threshold T;
(b) Creating a blank image with the size consistent with the dial area correction image and all pixel values of 0, drawing a pointer area on the blank image by using the vertex coordinates of the pointer rotation bounding box, and obtaining a mask image of the pointer area when the filled pixel value is 255;
(c) And superposing the mask image onto the dial area correction image, and performing threshold segmentation on the new image by using a threshold T to obtain a binary image only containing a pointer area.
Step 2.4) extracting pointer skeleton
(a) Carrying out corrosion operation in the belt direction on the binary image only containing the pointer region, eliminating adhesion among connected domains and ensuring the integrity of the pointer connected domain as far as possible, wherein the corrosion operation in the belt direction specifically comprises the following steps: firstly, respectively constructing structural elements in the directions of 0 °, 25 °, 45 °, 75 °, 90 °, 115 °, 135 °, and 160 ° as reference structural elements, wherein the distribution of the "1" element is at the above-mentioned angle, which is specifically as follows:
then, the rotation angle theta of the pointer in the step 2.3) is found in the above angles r Taking the structure element corresponding to the closest angle as a structure element for corrosion operation, and performing directional corrosion on the binary image only containing the pointer region obtained in the step 2.3);
(b) To the above-mentioned binary image mark connected domain that carries out the area direction corruption, obtain the coordinate list label _ dit of connected domain, for the adhesion of further avoiding connected domain, just judge same connected domain when satisfying four and adjoin between the pixel, the step of connected domain mark is: traversing each pixel point of the binary image point by point, wherein the label of the first point is set to be 1; when a new pixel point is processed, comparing the new pixel point with the left and upper points, and if the pixel value of the new pixel point is different from that of the left and upper points, adding 1 to the current label to be used as the label of the new pixel point; if the pixel value of the point is only the same as that of the left point or the upper pixel point, setting the label of the point as the label of the point with the same pixel value; if the pixel value of the point is the same as that of the left and upper points and the labels of the left and upper points are the same, setting the point as the same label; if the pixel values of the point are the same as the pixel values of the left and upper points and the labels of the left and upper points are different, setting the point as a label with a smaller label value, and changing the larger label value into the smaller label value;
(c) Finding a longest label _ fact [ len _ max ] in a connected domain coordinate list label _ fact, namely a point set contained in a maximum connected domain, and performing least square straight line fitting on all points in the point set to obtain a pointer skeleton, wherein the step of least square straight line fitting is as follows: assuming that the point set label _ fact [ len _ max ] satisfies the linear relation y = kx + c, where k, c are the slope and intercept, respectively, as long as a set of k, c is found, and the square deviation e of the point set label _ fact [ len _ max ] from this fitted straight line is minimized, the square deviation is expressed as:
wherein (x) i ,y i )∈label_dict[len_max]Then, the partial derivatives of k and c are calculated by the function e respectively, when
Then, the square deviation e can be minimized, and the linear equation system about k, c is obtained by arrangement:
solving the equation set of the first two-dimensional equation to obtain a point set label _ ditt [ len _ max [ ]]Y = kx + c, where n is label _ fact [ len _ max]Length of (d). Taking the fitted straight line to fall on the point set label _ ditt [ len _ max ]]The part in the minimum circumscribed rectangle is the pointer skeleton, and the coordinates of the pointer endpoint are recorded as
FIG. 5 shows the effect of partial dial image masking, threshold segmentation, directional erosion, and least squares line fitting to extract the pointer.
And (3) identifying the scale value, which comprises the following specific steps:
step 3.1) making a data set to train and test the EAST network
(a) Cutting a dial plate image from the collected instrument image in the same way as the second data set is made in the step 2), removing the background, and expanding the data set in a zooming, rotating and turning way to obtain a third data set;
(b) And labeling the third data set by adopting an image labeling tool Labelme to obtain a corresponding json file. All the scale value texts need to be marked, and the marking frame can be adjusted according to the direction of the texts. Then, sequentially extracting four vertex coordinates of the labeling frame from each json file
And calculating the rotation angle of the labeling frame by using the vertex coordinates: first, find the longest side of the label box, assuming its end point is
And rotating the positive half shaft of the x axis clockwise around the origin, wherein when the positive half shaft is parallel to the longest edge for the first time, the rotated angle is the rotation angle of the labeling frame. The rotation angle of the label box can be calculated by using the following formula:
(c) Dividing the dial plate images in the third data set and the corresponding txt label files into a training set and a testing set according to 4:1, inputting the training set into an EAST network for training, adjusting parameters of the network, wherein the parameters comprise learning rate, batch _ size and epoch, stopping training when loss does not decrease any more, and preventing overfitting; testing the network by using a test set; and finally, inputting the dial plate area correction image img obtained in the step 1.3) into the network to obtain the vertex coordinates and the rotation angle of the scale value text bounding box.
Step 3.2) intercepting the scale value text image and sending the scale value text image into a CRNN network for identification, wherein the specific steps are as follows: recording the vertex coordinate of the text bounding box with a certain scale value obtained in the step 3.1) as (p) 1 ,q 1 )、(p 2 ,q 2 )、(p 3 ,q 3 )、(p 4 ,q 4 ) At a rotation angle of β r Through a rotation angle of beta r Obtaining a rotation matrix R 2 Reuse of the rotation matrix R 2 Rotating and converting the dial plate area correction image, and rotating the scale value text to the horizontal direction; then, respectively matching the four vertex coordinates of the scale value text bounding box before rotation with the rotation matrix R 2 Dot-multiply to obtain the rotated coordinate (p) 1 ',q 1 ')、(p 2 ',q 2 ')、(p 3 ',q 3 ')、(p 4 ',q 4 ') to a host; finally, using the image interception method img [ q) in step 2.3) 3 ':q 1 ',p 3 ':p 1 '](or img [ q ] 4 ':q 2 ',p 4 ':p 2 ']) And intercepting the scale value text picture. Intercepting all scale value texts detected by the EAST network by adopting the method, and sequentially sending the texts into the CRNN network for identification to obtain scale value data on the dial plate. The scale recognition result of a part of the dial image is shown in fig. 6.
And (4) reading calculation, which comprises the following specific steps:
(a) And (3) performing circle fitting on the central point of the text bounding box of the dial scale value, and recording the center of the fitting circle as (p) c ,q c ) The radius is recorded as R, and the pointer endpoint obtained in the step 2.4) is calculated
Taking the end point with small distance as the tip of the pointer and recording as P;
(b) Sequentially calculating the distance from the central point of each scale value text bounding box to the pointer tip P, finding two points nearest to the pointer tip P, and recording the distance as d 1 、d 2 The corresponding scale value is denoted by k 1 、k 2 ；
(c) Calculating the point P of the pointer to the centre of the fitting circleDistance, noted as d p By using the cosine theorem, the scale k can be respectively calculated 1 、k 2 Angle theta with the pointer 1 、θ 2 ：
Calculating a reading k according to the angle proportional relation p ：
A schematic of the calculated readings is shown in fig. 7.
The invention totally tests 130 pictures containing different types of dial plates, and the test results are shown in tables 1 and 2:
TABLE 1 group of Dial plate picture reading identification results
| Serial number | Reading value | True value | Relative error | Rate of accuracy |
| 1 | 1.5323 | 1.519 | 0.8756％ | 99.1244％ |
| 2 | 0.4625 | 0.479 | 3.4447％ | 96.5553％ |
| 3 | 0.5625 | 0.565 | 0.4425％ | 99.5575％ |
| 4 | 0.6750 | 0.68 | 0.7353％ | 99.2647％ |
| 5 | 0.9250 | 0.922 | 0.3254％ | 99.6746％ |
| 6 | 1.1077 | 1.1 | 0.7000％ | 99.3000％ |
| 7 | 1.2688 | 1.26 | 0.6984％ | 99.3016％ |
| 8 | 20.9434 | 20.9 | 0.2077％ | 99.7923％ |
| 9 | 24.0476 | 24 | 0.1983％ | 99.8017％ |
| 10 | 8.1132 | 8 | 1.4150％ | 98.5850％ |
TABLE 2 statistics of reading recognition results (130 test pictures total)
As can be seen from Table 2, the average reading accuracy of the method can reach more than 98%, the identification precision is high, and the method has good practical application value.
Claims (7)
1. A pointer instrument automatic reading method based on rotating target detection is characterized by comprising the following steps:
step (1), dial area extraction and perspective correction are carried out to obtain a dial area correction image;
and (2) detecting and refining the pointer, which comprises the following specific steps:
step 2.1) construction and training based on R 3 The improved rotating target detection network of the Det obtains a pointer rotating bounding box through the rotating target detection network;
step 2.2) a mask of the pointer area is constructed according to the pointer rotation bounding box, the mask is superposed on the dial area correction image, and threshold segmentation is carried out to obtain a binary image only containing the pointer area;
step 2.3) carrying out corrosion with direction and connected domain marking on the binary image only containing the pointer region obtained in the step 2.2), and carrying out least square straight line fitting on the largest connected domain to obtain a pointer skeleton;
step (3), scale value identification;
and (4) reading calculation.
2. The pointer instrument automatic reading method based on rotating object detection as claimed in claim 1, characterized in that the specific steps of the step (1) are as follows:
step 1.1), inputting an original image into a trained Fast SCNN network to generate a dial area mask image, wherein the dial area mask image is a binary image, white in the mask image is a dial area predicted by the network, and black is a background;
step 1.2), carrying out ellipse detection on the dial area mask image to obtain a fitting ellipse of the dial outline, and respectively recording the central coordinate, the major semi-axis, the minor semi-axis and the inclination angle of the dial ellipse as (x) c ,y c )、a、b、θ；
Step 1.3), using the center as (x) c ,y c ) Intercepting an image of a dial area by a square frame with the side length of 2 a;
step 1.4), the central coordinates (x) of the ellipse of the dial are used c ,y c ) The long half shaft a, the short half shaft B and the inclination angle theta, and calculating the coordinates of four vertexes A, B, C, D, A ', B', C 'and D' of the dial ellipse before and after perspective transformation:
in the formula, the point A and the point B are the vertexes of the long half shaft and the short half shaft of the ellipse of the dial plate, the point C and the point D are the vertexes of the short half shaft of the ellipse of the dial plate, the perspective transformation matrix M is obtained by utilizing the coordinates of the vertexes before and after transformation, and then the dial plate area image and the perspective transformation matrix M are subjected to point multiplication to obtain the dial plate area correction image.
3. The method for automatically reading the pointer instrument based on the detection of the rotating target as claimed in claim 1, wherein the specific steps of the step 2.1) are as follows:
step 2.1.1), constructing and obtaining a rotating target detection network comprising an improved MobileNetv2 backbone network, an FPN characteristic pyramid structure, a characteristic refining module and a rotating target detector; the improved MobileNet v2 backbone network comprises the step of carrying out Batch Norm normalization processing on each convolution layer of a MobileNet v2 model; respectively leading out output characteristic diagrams of 3 rd, 4 th and 6 th bottomleneck structures of the MobileNet v2 model as input of an FPN structure in an original network; improving an inverse residual block of the MobileNet v2 network, adding an efficient attention module ECA-net after a Depth-wise convolution module, multiplying a weight vector output by the ECA-net and a Depth-wise convolution result in a channel dimension, and finally inputting the result to a subsequent Point-wise convolution module;
the improved MobileNet v2 backbone network is used for replacing an original ResNet backbone network and is used for extracting the image characteristics of a dial area; the method comprises the steps that the features extracted by a backbone network are acquired layer by the FPN feature pyramid structure, the features of different scales are fused, and the fused features are used as the input of a feature refining module; the feature refining module recodes the position information of the pointer bounding box and reconstructs the whole feature map to realize feature alignment, so that the detected pointer bounding box is more accurate; the rotating target detector realizes the detection of the bounding box with the direction and the classification of the target;
step 2.1.2), cutting out a dial plate image from the collected instrument image, removing a background, and expanding the data set in a zooming, rotating and overturning manner to obtain a second data set; labeling the second data set by adopting an image labeling tool Labelme to obtain a position label file of the pointer area, dividing the dial plate image and the corresponding position label file into a training set and a test set according to 4:1, inputting the training set into the rotating target detection network constructed in the step 2.1.1) for training, and testing the network by using the test set; inputting the dial plate area correction image obtained in the step 1.4) into a trained network to obtain a pointer rotation bounding box.
4. The method for automatically reading the pointer instrument based on the detection of the rotating target as claimed in claim 1, wherein the specific steps of the step 2.2) are as follows:
step 2.2.1), calculating the rotation angle theta by utilizing the vertex coordinates of the pointer rotation bounding box r The coordinates of the four vertexes are sequentially marked as (x) 1 ,y 1 )、(x 2 ,y 2 )、(x 3 ,y 3 )、(x 4 ,y 4 ) Calculating the angle of the longest side, and setting the vertex (x) 1 ,y 1 )、(x 2 ,y 2 ) The longest side, the rotation angle theta r The calculation formula of (2) is as follows:
through a rotation angle theta r Obtaining a rotation matrix R 1 Then using the rotation matrix R 1 Rotating and converting the dial area correction image, rotating the pointer to a horizontal position, intercepting the pointer area image by using an image slicing method, and then performing Otsu threshold segmentation on the pointer area image to obtain an optimal segmentation threshold T;
step 2.2.2), a blank image with the size consistent with the dial area correction image and all pixel values of 0 is created, a pointer area is drawn on the blank image by using the vertex coordinates of the pointer rotation bounding box, and the filled pixel value is 255, so that a mask image of the pointer area can be obtained;
and 2.2.3) overlapping the mask image to the dial area correction image, and performing threshold segmentation on the new image by using a threshold T to obtain a binary image only containing the pointer area.
5. The method for automatically reading the pointer instrument based on the detection of the rotating target as claimed in claim 1, wherein the specific steps of the step 2.3) are as follows:
step 2.3.1), respectively constructing structure elements in the directions of 0 degrees, 25 degrees, 45 degrees, 75 degrees, 90 degrees, 115 degrees, 135 degrees and 160 degrees, and rotating the structure elements by a rotation angle theta of a boundary box with the pointer r Taking the structure element closest to the angle as a structure element for corrosion operation, and performing corrosion with the direction on the binary image which only contains the pointer region and is obtained in the step 2.2);
step 2.3.2), marking the connected domain of the binary image subjected to the strip-direction corrosion, obtaining a coordinate list label _ dit of the connected domain, finding out the longest item in the label _ dit, namely the point set contained in the maximum connected domain, performing least square straight line fitting on all points in the point set to obtain a pointer skeleton, and marking the coordinates of the endpoint of the pointer as the coordinates of the pointer
6. The method for automatically reading the pointer instrument based on the detection of the rotating target as claimed in claim 5, wherein the specific steps of the step (3) are as follows: firstly, correcting images in a dial area, inputting the corrected images into a trained EAST text detection network to obtain coordinates of scale value text bounding boxes on the dial, and then sequentially intercepting scale value text images according to the coordinates and inputting the scale value text images into a CRNN network for recognition to obtain each scale value.
7. The method for automatically reading the pointer instrument based on the detection of the rotating target as claimed in claim 6, wherein the specific steps of the step (4) are as follows: firstly, performing circle fitting on the central point of the text bounding box of the dial scale value, and recording the circle center of the fitting as (p) c ,q c ) The radius is recorded as R), and the pointer endpoint obtained in the step 2.3) is calculated
Taking the end point with small distance as the distance from the fitting circle centerIs a pointer tip and is marked as P; then, sequentially calculating the distance from the central point of each scale value text bounding box to the tip P, finding two points closest to P, and recording the distance as d 1 、d 2 The corresponding scale value is recorded as k 1 、k 2 (ii) a Then, the distance from P to the center of the fitting circle is calculated and recorded as d P By using the cosine theorem, the scale k can be respectively calculated 1 、k 2 Angle theta with the pointer 1 、θ 2 ：
Finally, calculating the reading k according to the angle proportional relation p ：
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210807428.1A CN115205858A (en) | 2022-07-07 | 2022-07-07 | Pointer type instrument automatic reading method based on rotating target detection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210807428.1A CN115205858A (en) | 2022-07-07 | 2022-07-07 | Pointer type instrument automatic reading method based on rotating target detection |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115205858A true CN115205858A (en) | 2022-10-18 |
Family
ID=83580200
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210807428.1A Pending CN115205858A (en) | 2022-07-07 | 2022-07-07 | Pointer type instrument automatic reading method based on rotating target detection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115205858A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116012828A (en) * | 2022-12-02 | 2023-04-25 | 长扬科技(北京)股份有限公司 | Pointer instrument identification method and device, electronic equipment and storage medium |
| CN116363688A (en) * | 2023-03-23 | 2023-06-30 | 嘉洋智慧安全科技(北京)股份有限公司 | Image processing method, device, equipment, medium and product |
| CN116704510A (en) * | 2023-08-08 | 2023-09-05 | 深圳中集智能科技有限公司 | Intelligent identification method for machine vision industrial instrument data |
| CN116844058A (en) * | 2023-08-30 | 2023-10-03 | 广州市扬新技术研究有限责任公司 | Pointer instrument indication recognition method, device, equipment and storage medium |
| CN117372937A (en) * | 2023-12-07 | 2024-01-09 | 江西理工大学南昌校区 | Data reading method based on pointer instrument |
-
2022
- 2022-07-07 CN CN202210807428.1A patent/CN115205858A/en active Pending
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116012828A (en) * | 2022-12-02 | 2023-04-25 | 长扬科技(北京)股份有限公司 | Pointer instrument identification method and device, electronic equipment and storage medium |
| CN116012828B (en) * | 2022-12-02 | 2023-07-25 | 长扬科技(北京)股份有限公司 | Pointer instrument identification method and device, electronic equipment and storage medium |
| CN116363688A (en) * | 2023-03-23 | 2023-06-30 | 嘉洋智慧安全科技(北京)股份有限公司 | Image processing method, device, equipment, medium and product |
| CN116704510A (en) * | 2023-08-08 | 2023-09-05 | 深圳中集智能科技有限公司 | Intelligent identification method for machine vision industrial instrument data |
| CN116844058A (en) * | 2023-08-30 | 2023-10-03 | 广州市扬新技术研究有限责任公司 | Pointer instrument indication recognition method, device, equipment and storage medium |
| CN116844058B (en) * | 2023-08-30 | 2024-03-12 | 广州市扬新技术研究有限责任公司 | Pointer instrument indication recognition method, device, equipment and storage medium |
| CN117372937A (en) * | 2023-12-07 | 2024-01-09 | 江西理工大学南昌校区 | Data reading method based on pointer instrument |
| CN117372937B (en) * | 2023-12-07 | 2024-03-29 | 江西理工大学南昌校区 | Data reading method based on pointer instrument |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115205858A (en) | Pointer type instrument automatic reading method based on rotating target detection | |
| CN108355987B (en) | A kind of screen printing of battery quality determining method based on piecemeal template matching | |
| CN111414934A (en) | Pointer type meter reading automatic identification method based on fast R-CNN and U-Net | |
| JP5379085B2 (en) | Method and system for classifying connected groups of foreground pixels in a scanned document image based on marking type | |
| US8654398B2 (en) | Method for simulating impact printer output, evaluating print quality, and creating teaching print samples | |
| CN112149667B (en) | Automatic reading method of pointer instrument based on deep learning | |
| CN105574527A (en) | Quick object detection method based on local feature learning | |
| CN108961230B (en) | Identification and extraction method for structural surface crack characteristics | |
| Chen et al. | Shadow-based Building Detection and Segmentation in High-resolution Remote Sensing Image. | |
| CN108629762B (en) | Image preprocessing method and system for reducing interference characteristics of bone age evaluation model | |
| CN103439348A (en) | Remote controller key defect detection method based on difference image method | |
| CN116168037B (en) | Method and system for calculating bending degree of wire crimping based on image processing | |
| CN112734729A (en) | Water gauge water level line image detection method and device suitable for night light supplement condition and storage medium | |
| CN112489026B (en) | Asphalt pavement disease detection method based on multi-branch parallel convolution neural network | |
| CN108664970A (en) | A kind of fast target detection method, electronic equipment, storage medium and system | |
| CN115393861B (en) | Method for accurately segmenting handwritten text | |
| CN112364834A (en) | Form identification restoration method based on deep learning and image processing | |
| CN107388991A (en) | A kind of more fillet axial workpiece radius of corner measuring methods in end face | |
| CN117094916A (en) | Visual inspection method for municipal bridge support | |
| CN115546795A (en) | Automatic reading method of circular pointer instrument based on deep learning | |
| CN113435442B (en) | Water level measuring method and device, water gauge and electronic equipment | |
| CN111553345B (en) | Method for realizing meter pointer reading identification processing based on Mask RCNN and orthogonal linear regression | |
| CN114359300B (en) | Optimization method, device and system of image segmentation model and storage medium | |
| CN110377640A (en) | One kind is based on deep learning ballot statistical method | |
| CN115527067A (en) | Method, system and storage medium for verifying contents of image-text printed matter |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |