CN115019294A - Pointer instrument reading identification method and system - Google Patents
Pointer instrument reading identification method and system Download PDFInfo
- Publication number
- CN115019294A CN115019294A CN202210706467.2A CN202210706467A CN115019294A CN 115019294 A CN115019294 A CN 115019294A CN 202210706467 A CN202210706467 A CN 202210706467A CN 115019294 A CN115019294 A CN 115019294A
- Authority
- CN
- China
- Prior art keywords
- pointer instrument
- pointer
- scale value
- key point
- instrument
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/02—Recognising information on displays, dials, clocks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
A reading identification method and a reading identification system for a pointer instrument relate to the technical field of computer vision and comprise the following steps: s1, acquiring a pointer instrument image, and constructing a pointer instrument image data set; s2, constructing a pointer instrument scale value detection network model, and training the pointer instrument scale value detection network model by using a pointer instrument image data set; s3, constructing a pointer instrument key point detection network model, and training the pointer instrument key point detection network model by using a pointer instrument image data set; s4, carrying out scale value prediction on the pointer instrument image to be recognized by using the trained pointer instrument scale value detection network model; s5, performing key point prediction on the pointer instrument image to be recognized by using the trained pointer instrument key point detection network model; and S6, performing post-processing analysis on the prediction result according to the scale value of the pointer instrument and the prediction result of the key point, and acquiring the current reading of the pointer instrument.
Description
Technical Field
The invention relates to a reading identification method and system for a pointer instrument, and belongs to the technical field of computer vision.
Background
The pointer type instrument is one of the most common measuring instruments in the current production process, and not only has a great variety, but also has a huge number. At present, the manual observation instrument dial pointer reading is mostly adopted for the verification work of the instruments in China, and the traditional method is influenced by other subjective factors such as the operation experience, the working attitude, the service level and the mental state of detection personnel, so that the problems of high labor intensity, low production efficiency, large detection error, poor reliability and the like exist. In addition, under severe working environments such as high temperature, radiation, field and the like, the method is not suitable for manually monitoring the reading of the pointer instrument, and under the condition, the image identification is increasingly applied.
Chinese patent document CN202010326587 discloses an SF6 meter image reading method based on HRNet network model, wherein, the processing of data set is to label the scale key points, the pointers, the rotation center points of the pointers on the meter and the middle intersection points of the SF6 pointer table, then the adopted key point detection algorithm is top-down HRNet, that is, each key point is predicted by using a feature map, the different feature maps in the document predict the key point information of the specified position according to the sequence, which requires labeling according to a certain sequence strictly when labeling, and for the prediction result of the key point, the document only aims at one pointer phenotype, so the reading processing is performed in combination with the priori knowledge of scale values, and the adopted reading method is an angle discrimination method.
Chinese patent document CN202010326587 has the following characteristics: in a data set processing mode, the marked information comprises scale key points, pointers, rotating central points of the pointers on an instrument and middle intersection points, and strict marking sequence requirements are met; (ii) using an algorithm: the HRNet used in this document is a top-down keypoint detection algorithm, and uses one feature map to predict each keypoint on a picture, and as described in the above patent, a total of 14 feature maps are required to complete the prediction of all keypoints. Integral processing logic: the document only adopts a key point detection algorithm and the prior knowledge of an SF6 pointer table to carry out reading detection, and the situations of wrong detection and missed detection of a model detection result cannot be corrected. Application range: the document only processes the SF6 pointer table, and a large amount of prior knowledge is combined to finally complete reading, so that the document cannot be applied to all types of pointer tables, and the generalization is poor. The post-treatment method comprises the following steps: the post-processing method of the document is to perform reading by combining the prior ratio between the angles, and the reliability and the confidence of the result are to be improved.
Chinese patent document CN202111422893 discloses an instrument reading identification method and device based on machine vision: firstly, detecting a meter outer frame in a picture by using a yolov5 target detection algorithm, then cutting a detected meter image to determine the position of a pointer and the position information of a segmented character by using the gray scale, threshold segmentation, minimum binarization and other methods of the traditional image processing method, then sending the segmented character image into a resnet classification algorithm for classification, and finally combining the processing result of the previous step to perform reading processing.
Chinese patent document CN202111422893 has the following characteristics: algorithm logic: in the document, a meter image is detected by using a target detection algorithm, then pointer and scale value information are obtained by using a traditional image processing algorithm, and finally a scale value result is obtained by using a classification algorithm, so that the reading algorithm is finally obtained. Detecting the target by a target detection algorithm: the document uses a target detection algorithm only for detecting the position and the type of the image of the meter to obtain the information of the whole meter. ③ stability of the algorithm: in the document, because the traditional image processing algorithm is adopted to obtain the information of the pointer and the information of the scale value, the traditional image processing algorithm has great limitation, has high requirement on the quality of the image, and is easily influenced by external factors such as illumination and the like, so that the whole reading algorithm is unstable. The application range is as follows: due to the limitation of the method, the method can not be applied to all kinds of meter reading, and only can be used for reading identification of one or more meter types with few dial interference factors.
In summary, most of the conventional algorithms operate on a picture which is manually preprocessed (for example, a dial is put in the middle of the picture and is enlarged to the whole picture, and a surrounding background is removed, etc.) aiming at a simpler situation, and only a pointer with a specific shape can be recognized, which cannot well meet the current working requirement.
Disclosure of Invention
Aiming at the defects of the prior art, the invention discloses a pointer type instrument reading identification method.
The invention also discloses a system for loading the identification method, which is used for reading the pointer instrument.
The detailed technical scheme of the invention is as follows:
a reading identification method for a pointer instrument is characterized by comprising the following steps:
s1, constructing a pointer instrument image data set: respectively labeling the pointer instrument images to correspondingly form a labeled pointer instrument scale value data set and a labeled pointer instrument key point data set;
s2, constructing a pointer instrument scale value detection network model: training a pointer instrument scale value detection network model by using the marked pointer instrument scale value data set;
constructing a key point detection network model of the pointer instrument: training a pointer instrument key point detection network model by using the marked pointer instrument key point data set;
s3, carrying out scale value prediction on the pointer instrument image to be recognized by using the trained pointer instrument scale value detection network model;
carrying out key point prediction on a pointer instrument image to be recognized by utilizing a trained pointer instrument key point detection network model;
and S4, performing post-processing analysis according to the prediction result of the step S3, and acquiring the current pointer instrument reading in the pointer instrument diagram to be recognized.
Preferably, the constructing of the pointer instrument image data set in step S1 specifically includes:
s11, shooting a pointer instrument image by using a camera;
s12, acquiring videos shot by each camera, and collecting video streams at different time periods;
s13, processing the collected video stream by adopting a frame extraction method to obtain a plurality of pictures; the frame extracting time interval is 30min, and one picture is extracted from the video every 30 min;
s14, summarizing all collected pictures, and screening all the pictures containing pointer instrument images to produce an unlabelled pointer instrument image data set;
s15, marking pointer type instrument image data set:
labeling dial plate scale values in pictures of the image data set of the unmarked pointer instrument by using an open source labeling tool Labelimg, wherein the labeling type is the dial plate scale value, forming a scale value data set of the pointer instrument with completed labeling, and the scale value data set of the pointer instrument with completed labeling is according to 7: 2: 1, distributing the training set, the test set and the verification set out of order;
marking dial scales, pointer needles and pointer tails in pictures of the image data set of the unmarked pointer instrument by using an open source marking tool Labelme to form a marked key point data set of the pointer instrument, and marking the marked key point data set of the pointer instrument according to the following steps of: 2: 1, out-of-order into a training set, a test set, and a validation set.
Preferably, in step S2, the pointer instrument scale value detection network model is built by using a modified yolov5 target detection algorithm, and includes: adding an ASFF self-adaptive feature fusion module in a yolov5 target detection algorithm, and performing weighted fusion on an upper output feature layer;
the ASFF adaptive feature fusion module adopts a formula:
input features of upper layer
Multiplying the weight parameters respectively
Obtaining the feature fusion map of the next layer
Weight parameter
After dimension reduction, the softmax function makes the range of the softmax function be 0,1]Internal and the sum is 1; the subscript and the superscript in formula (I) represent the correspondence of the different feature fusion maps before and after.
Preferably, in step S2, the pointer instrument key point detection network model adopts a modified bottom-up hrnet key point detection algorithm: in step S15, the dial scale, the pointer needle head, and the pointer needle tail marked by the label are respectively a pointer instrument scale feature layer, a pointer instrument needle head feature layer, and a pointer instrument needle tail feature layer;
for a pointer instrument scale feature layer, extracting a pointer instrument scale key point by using an associated Embedding post-processing method; extracting the position of the maximum characteristic value in the characteristic diagram as the needle head key point and the needle tail key point of the pointer instrument for the needle head characteristic layer and the needle tail characteristic layer of the pointer instrument; wherein, the associated Embedding post-processing method is a post-processing method provided in a paper of associated Embedding, End-to-End Learning for Joint Detection and Grouping; extracting the maximum characteristic value in the characteristic diagram: the characteristic diagram refers to a characteristic result diagram obtained by extracting characteristics from an image by a neural network, the characteristic diagram refers to a two-dimensional digital matrix, and the maximum characteristic value is the maximum numerical value in the digital matrix.
Preferably, in step S3, the predicting the scale value of the pointer instrument image to be recognized by using the trained pointer instrument scale value detection network model specifically includes:
inputting the pointer instrument image to be identified into a trained pointer instrument scale value detection network model for feature extraction, and performing self-adaptive feature fusion on different output layers according to the extracted features:
and detecting the position and the type of the scale value on the dial plate in the pointer instrument image to obtain the position and the type information of the scale value on the dial plate in the pointer instrument image. Wherein the different output layers refer to: the neural network comprises a plurality of network layers, wherein an output layer refers to intermediate network layers with different sizes extracted by the network; the category is the number represented by the scale value on the dial, for example, if the scale value is 1.0, the category corresponding to the scale value is 1.0. The fused network can better extract features in the technical scheme, and the feature fusion method is adopted to enable the network to have better effect and more accurate detection.
Preferably, in step S3, the performing the keypoint prediction on the pointer instrument image to be recognized by using the trained pointer instrument keypoint detection network model specifically includes:
inputting the pointer instrument image to be identified into a trained pointer instrument key point detection network model for feature extraction, and directly obtaining the key point information through associated Embedding post-processing according to the extracted features:
and detecting the positions and the types of the dial scales, the pointer needle heads and the pointer tails in the pointer instrument image to obtain the positions and the type information of the dial scales, the pointer needle heads and the pointer tails in the pointer instrument image, wherein the types are the three types of the dial scales, the pointer needle heads and the pointer tails.
Preferably, the step S4 includes the following steps:
s41, aiming at each scale value prediction result of the scale value detection model of the pointer instrument, judging the position of the scale value according to the clockwise arrangement and the principle of following from small to large:
if the position of the scale value follows the principle, the prediction result of the scale value is correct;
if the position of the scale value does not conform to the principle, the scale value prediction result is wrong, and the scale value prediction result is rejected;
then traversing each prediction result of the pointer instrument key point detection model, and respectively calculating the position of the scale value and the distance of each dial scale key point: taking the key point with the minimum distance to correspond to the position of the scale value, namely the scale value represented by the key point;
s42, after all the scale values are in one-to-one correspondence with the key points, traversing all the key points according to the clockwise direction:
if there is a key point without corresponding scale value, but the left and right key points have scale values, the scale value of the left key point is set as S n-1 The scale value of the right key point is S n+1 Then the scale value S of the key point n ＝(S n-1 +S n+1 )/2；
After traversing for one time, if the key points which do not correspond to the scale values exist, deleting the key point information;
s43, traversing all the remaining key points with corresponding scale values, calculating the minimum distance from the key point position to the needle head and the needle tail line segment of the pointer, and finding out the key points on two sides of the pointer:
let Sd be the minimum distance from the key point on the left side of the pointer to the pointer n-1 Scale value of S n-1 (ii) a The minimum distance from the key point on the right side of the pointer to the pointer is Sd n+1 Scale value of S n+1 (ii) a The reading of the current pointer instrument is Sn ═ S (S) n+1 +Sd n+1 *S n-1 /Sd n-1 )/(1+Sd n+1 /Sd n-1 ) Obtaining the reading result of the current pointer table, wherein the rest of the reading result refers to the remaining key points with the corresponding scale values after the key points without the corresponding scale values are deleted; the minimum distance from the key point to the needle head and the needle tail line segment of the pointer is calculated, wherein the minimum distance from the key point to the line segment of the needle head needle tail connecting line is calculated, and part of the minimum distance is the vertical line and part of the minimum distance is the connecting line distance from the key point to the line segment vertex.
A system for loading the recognition method, comprising:
the device comprises an image acquisition module, a prediction module loaded with a pointer instrument scale value detection network model and a pointer instrument key point detection network model, and a post-processing analysis module to obtain the current pointer instrument reading in a pointer instrument graph to be identified;
the image acquisition module is loaded with the method of the step S1;
the prediction module loaded with the pointer instrument scale value detection network model and the pointer instrument key point detection network model is used for processing according to the methods of the steps S2 and S3;
the post-processing analysis module is loaded with the method of step S4.
The technical advantages of the invention include:
1. the invention designs a pointer instrument reading identification method and a pointer instrument reading identification system, provides an improved yolov5 target detection algorithm, adds an ASFF adaptive feature fusion module in the existing yolov5 target detection algorithm, detects and identifies pointer instrument images, and improves the accuracy and the real-time performance of the algorithm on target detection and identification.
2. The invention combines the target detection algorithm and the key point detection algorithm, provides a method taking the key point detection algorithm as the main part and taking the target detection algorithm as the auxiliary part, can identify the position and the category information of the dial scale, the pointer needle head and the pointer tail, can also identify the position and the category information of the scale value on the dial, and combines the identification results of the two algorithm models to read the pointer instrument, thereby greatly increasing the accuracy and the reliability of the reading algorithm.
3. The invention designs a post-processing method based on scale value detection network model of a pointer instrument and key point detection network model results of the pointer instrument, and provides a new reading calculation method.
Drawings
FIG. 1 is a flow chart of a method for identifying reading of a pointer instrument according to the present invention;
FIG. 2 is a schematic diagram of a scale value data set of a dial plate of the pointer instrument marked with scale values;
FIG. 3 is a schematic diagram of a pointer instrument key point data set for key point labeling;
FIG. 4 is a schematic diagram of a detection result of a needle-type instrument scale value detection network model;
FIG. 5 is a schematic diagram of a detection result of a needle-type instrument key point detection network model;
FIG. 6 is a schematic representation of the identification of pointer meter readings using the method of the present invention.
Detailed Description
The present invention is further described, but not limited to, in the following figures:
the method is applied to a certain station for monitoring and shooting the video of the pointer instrument installed on the wall in real time. After a plurality of pictures are extracted from a shot video stream at certain time intervals and sent into a target detection model (the target detection model is a pointer instrument scale value detection network model) and a key point detection model (the key point detection model is a pointer instrument key point detection network model) adopted by the invention, the target detection model detects target information at a certain moment. The target information comprises the position and the category information of the scale value on the dial plate. The key point detection model detects key point information at a certain moment. The key point information comprises dial scales, a pointer head and position and category information of a pointer tail. And analyzing the detected target information and the key point information, interpreting the pointer reading information at the current moment, and acquiring the pointer reading information in real time.
Examples 1,
As shown in fig. 1, a reading identification method for a pointer instrument includes:
s1, constructing a pointer instrument image data set: respectively labeling the pointer instrument images to correspondingly form a labeled pointer instrument scale value data set and a labeled pointer instrument key point data set;
s2, constructing a pointer instrument scale value detection network model: training a pointer instrument scale value detection network model by using the marked pointer instrument scale value data set;
constructing a key point detection network model of the pointer instrument: training a pointer instrument key point detection network model by using the marked pointer instrument key point data set;
s3, carrying out scale value prediction on the pointer instrument image to be recognized by using the trained pointer instrument scale value detection network model;
carrying out key point prediction on a pointer instrument image to be recognized by utilizing a trained pointer instrument key point detection network model;
and S4, performing post-processing analysis according to the prediction result of the step S3, and acquiring the current pointer instrument reading in the pointer instrument diagram to be recognized.
The constructing of the pointer instrument image data set in step S1 specifically includes:
in the embodiment, 2724 pictures are total in the collected pointer instrument image data;
s11, shooting a pointer instrument image by using a camera;
s12, acquiring videos shot by each camera, and collecting video streams at different time periods;
s13, processing the collected video stream by adopting a frame extraction method to obtain a plurality of pictures; the frame extracting time interval is 30min, and one picture is extracted from the video every 30 min;
s14, summarizing all collected pictures, and screening all the pictures containing pointer instrument images to produce an unlabelled pointer instrument image data set;
s15, marking pointer type instrument image data set:
as shown in fig. 2, an open source labeling tool Labelimg is used to label dial scale values in each picture of an image data set of an unlabeled pointer instrument, the labeling type is the dial scale value, a scale value data set of the labeled pointer instrument is formed, and the scale value data set of the labeled pointer instrument is calculated according to the following formula: 2: 1, distributing the training set, the test set and the verification set out of order;
as shown in fig. 3, an open source labeling tool Labelme is used to label dial scales, a pointer needle head and a pointer needle tail in each picture of an image data set of an unlabelled pointer instrument to form a labeled pointer instrument key point data set, and the labeled pointer instrument key point data set is represented by 7: 2: 1, out-of-order into a training set, a test set, and a validation set.
In step S2, the pointer instrument scale value detection network model is built by using an improved yolov5 target detection algorithm, and includes: adding an ASFF self-adaptive feature fusion module in a yolov5 target detection algorithm, and performing weighted fusion on an upper output feature layer;
the ASFF adaptive feature fusion module adopts a formula:
input features of upper layer
Multiplying the weight parameters respectively
Obtaining the feature fusion map of the next layer
Weight parameter
After dimension reduction, the ranges of the two are all [0,1 ] through a softmax function]Internal and the sum is 1; the subscript and the superscript in formula (I) represent the correspondence of the different feature fusion maps before and after.
In step S2, the pointer instrument key point detection network model adopts an improved bottom-up hrnet key point detection algorithm: in step S15, the dial scale, the pointer needle head, and the pointer needle tail marked by the label are respectively a pointer instrument scale feature layer, a pointer instrument needle head feature layer, and a pointer instrument needle tail feature layer;
for a pointer instrument scale feature layer, extracting a pointer instrument scale key point by using an associated Embedding post-processing method; extracting the position of the maximum characteristic value in the characteristic diagram as the needle head key point and the needle tail key point of the pointer instrument for the needle head characteristic layer and the needle tail characteristic layer of the pointer instrument; wherein, the associated Embedding post-processing method is a post-processing method provided in a paper of associated Embedding, End-to-End Learning for Joint Detection and Grouping; extracting the maximum characteristic value in the characteristic diagram: the characteristic diagram refers to a characteristic result diagram obtained by extracting characteristics from an image by a neural network, the characteristic diagram refers to a two-dimensional digital matrix, and the maximum characteristic value is the maximum numerical value in the digital matrix.
Examples 2,
In step S3, as shown in fig. 4, the method for predicting scale values of a pointer instrument image to be recognized by using a trained pointer instrument scale value detection network model specifically includes:
inputting the pointer instrument image to be identified into a trained pointer instrument scale value detection network model for feature extraction, and performing self-adaptive feature fusion on different output layers according to the extracted features:
and detecting the position and the type of the scale value on the dial plate in the pointer instrument image to obtain the position and the type information of the scale value on the dial plate in the pointer instrument image. Wherein the different output layers refer to: the neural network comprises a plurality of network layers, wherein an output layer refers to a middle network layer with different sizes obtained by network extraction; the category is the number represented by the scale value on the dial, for example, if the scale value is 1.0, the category corresponding to the scale value is 1.0. The network can better extract the characteristics after the fusion in the technical scheme, and the characteristic fusion method is adopted to ensure that the network has better effect and the detection is more accurate.
In step S3, as shown in fig. 5, the predicting key points of the pointer instrument image to be recognized by using the trained pointer instrument key point detection network model specifically includes:
inputting the pointer instrument image to be identified into a trained pointer instrument key point detection network model for feature extraction, and directly obtaining the key point information through associated Embedding post-processing according to the extracted features:
and detecting the positions and the types of the dial scales, the pointer needle heads and the pointer tails in the pointer instrument image to obtain the positions and the type information of the dial scales, the pointer needle heads and the pointer tails in the pointer instrument image, wherein the types are the three types of the dial scales, the pointer needle heads and the pointer tails.
The pointer instrument key point detection network model adopts an improved bottom-to-top hrnet key point detection algorithm, the bottom-to-top hrnet key point detection algorithm is improved, different processing methods are adopted for different feature layers of the algorithm, for a pointer instrument scale feature layer, a pointer instrument scale key point is extracted by using an associated Embedding post-processing method, and for a pointer instrument pinhead feature layer and a pointer instrument pintail feature layer, the position of the maximum feature value in a feature map is extracted to serve as the pointer instrument pinhead key point and the pointer instrument pintail key point. And the improved bottom-to-top hrnet key point detection algorithm does not need subsequent clustering processing, and only needs to obtain final key point information according to the feature map, so that the model reasoning time is greatly shortened, and the real-time performance of the key point model is improved.
Examples 3,
According to the method of embodiment 2, the specific step of step S4 includes:
s41, aiming at each scale value prediction result of the scale value detection model of the pointer instrument, judging the position of the scale value according to the clockwise arrangement and the principle of following from small to large:
if the position of the scale value follows the principle, the prediction result of the scale value is correct;
if the position of the scale value does not conform to the principle, the scale value prediction result is wrong, and the scale value prediction result is rejected;
then traversing each prediction result of the pointer instrument key point detection model, and respectively calculating the position of the scale value and the distance of each dial scale key point: taking the key point with the minimum distance to correspond to the position of the scale value, namely the scale value represented by the key point;
s42, after all the scale values are in one-to-one correspondence with the key points, traversing all the key points according to the clockwise direction:
if there is a key point without corresponding scale value, but the left and right key points have scale values, the scale value of the left key point is set as S n-1 The scale value of the right key point is S n+1 Then the scale value S of the key point n ＝(S n-1 +S n+1 )/2；
After one-time traversal, if the key points which do not correspond to the scale values exist, deleting the key point information;
s43, traversing all the remaining key points with corresponding scale values, calculating the minimum distance from the key point position to the needle head and the needle tail line segment of the pointer, and finding out the key points on two sides of the pointer:
let Sd be the minimum distance from the key point on the left side of the pointer to the pointer n-1 Scale value of S n-1 (ii) a The minimum distance from the key point on the right side of the pointer to the pointer is Sd n+1 Scale value of S n+1 (ii) a The reading of the current pointer instrument is Sn ═ S (S) n+1 +Sd n+1 *S n-1 /Sd n-1 )/(1+Sd n+1 /Sd n-1 ) Obtaining the reading result of the current pointer table, as shown in fig. 6 specifically, where the rest of the key points refers to the key points with corresponding scale values left after the key points without corresponding scale values are deleted; the minimum distance from the key point to the needle head and the needle tail line segment of the pointer is calculated, wherein the minimum distance from the key point to the line segment of the needle head needle tail connecting line is calculated, and part of the minimum distance is the vertical line and part of the minimum distance is the connecting line distance from the key point to the line segment vertex.
As shown in fig. 6, firstly, the scale key point detection result and the scale value detection result are in one-to-one correspondence, that is, the key points other than the needle tail key point correspond to the scale value detection frames, as can be seen from the figure, according to the distance discrimination principle, there are 5 key points of 0, 0.4, 0.8, 1.2 and 1.6 which can be directly corresponded, and then, by judging whether the key points on both sides of the key point which does not correspond to the scale value have the corresponding scale value, S is utilized to determine whether the key points on both sides of the key point which does not correspond to the scale value have the corresponding scale value n ＝(S n-1 +S n+1 ) The calculation method of/2 obtains the scale value corresponding to the key point, for example: the key point between 0 and 0.4 has no corresponding scale value information, according to the formula, S is (0+0.4)/2 is 0.2, that is, the scale value corresponding to the key point is 0.2, other key points are similar, finally, the scale values of 0, 0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4 and 1.6 can be obtained, then, the distances from the key points corresponding to the scale values to the needle tail line segment of the needle head are calculated, that is, the distance between 0.4 and 0.6 and the line segment is the minimum, that is, the distance Sd is obtained between the two scales respectively 0.4 And Sd 0.6 Two minimum distances, then openFormula Sn ═ S n+1 +Sd n+1 *S n-1 /Sd n-1 )/(1+Sd n+1 /Sd n-1 ) Where Sn-1 is 0.4 and Sn +1 is 0.6, the final reading Sn is 0.5532. The reading results of the image shown are shown on the upper left side of fig. 6.
Examples 4,
A system for loading the identification method according to embodiments 1, 2 and 3, comprising:
the device comprises an image acquisition module, a prediction module loaded with a pointer instrument scale value detection network model and a pointer instrument key point detection network model, and a post-processing analysis module to obtain the current pointer instrument reading in a pointer instrument graph to be identified;
the image acquisition module is loaded with the method of the step S1;
the prediction module loaded with the pointer instrument scale value detection network model and the pointer instrument key point detection network model is used for processing according to the methods of the steps S2 and S3;
the post-processing analysis module is loaded with the method of step S4.
In the embodiment, an improved yolov5 target detection algorithm is used for identifying dial scale values and an improved bottom-to-top hrnet key point detection algorithm is used for identifying key point information of dial scales, a pointer needle head and a pointer needle tail so as to obtain the reading of the pointer instrument at the current moment. The dial scale value and the dial scale, the pointer needle head and the pointer tail are processed under two conditions, the category and the position of the dial scale value and the category and the position information of key points of the dial scale, the pointer needle head and the pointer tail are respectively detected, then the detection results are integrated, and corresponding post-processing reading is carried out, so that the accurate reading of the pointer instrument is realized, and the accurate reading of the pointer instrument can meet the requirements of pointer instruments of various types, particularly the pointer instruments such as a sulfur hexafluoride meter, an ammeter, a voltmeter, an oil level meter, a lightning arrester meter, a pressure gauge, an oil temperature meter, a respirator and a pressure regulating box.
The invention needs to use a large amount of marked pointer instrument image data for pre-training, and can apply the algorithm to the reading of the unknown pointer instrument on the basis of the pre-training, so the invention mainly comprises two parts: the technical key points of the deep learning algorithm training and predicting stage and the algorithm post-processing stage are that the problems of high labor intensity, low production efficiency, large detection error, poor reliability and the like of the reading of a dial pointer of an artificial observation instrument and the problems of being influenced by severe working environments such as high temperature, radiation, field and the like can be solved, and the identification flexibility and the identification precision are improved while the identification cost of the reading of a meter is reduced.
In summary, compared with the prior art, the invention has the following advantages:
(1) the invention combines the detection result of the target detection algorithm on the scale value of the image of the pointer instrument and the detection result information of the key point detection algorithm on the scale value of the image dial plate, the needle head and the needle tail of the pointer instrument, and realizes the reading function of the current pointer instrument through a series of post-processing algorithms. The invention fully utilizes the information on the dial plate, constructs the relation of key points of clockwise scales through the detected result, can correct and supplement the missing missed scale value and the false detection wrong scale value, finally obtains correct scale value distribution, and then obtains the final reading of the pointer instrument according to the relation between the scale value and the needle head and the needle tail of the pointer.
(2) The invention does not need any prior knowledge, is also suitable for the small-amplitude water level gauge inclination condition, and does not influence the final reading. The invention determines the position and reading of dial scale value by improved yolov5 target detection algorithm and determines the category and position information of dial scale, pointer needle and pointer needle tail by improved bottom-up hrnet key point detection algorithm, thus realizing accurate reading of pointer instrument with error within 2 small scales more than 90%.
(3) The invention does not need to correct the inclination of the pointer instrument, and can be directly applied to the condition that the dial can be seen clearly when the pointer instrument is inclined, and no redundant processing is needed.
(4) The invention can satisfy various types of pointer type instruments, can carry out reading identification as long as the dial plate is provided with scale values, scales, pointers and the like, and has wide application range.
The above-mentioned embodiments are merely illustrative of the preferred embodiments of the present invention, and do not limit the scope of the present invention, and various modifications and improvements of the technical solution of the present invention by those skilled in the art should fall within the protection scope defined by the claims of the present invention without departing from the spirit of the present invention.
Claims (8)
1. A reading identification method for a pointer instrument is characterized by comprising the following steps:
s1, constructing a pointer instrument image data set: respectively labeling the pointer instrument images to correspondingly form a labeled pointer instrument scale value data set and a labeled pointer instrument key point data set;
s2, constructing a pointer instrument scale value detection network model: training a pointer instrument scale value detection network model by using the marked pointer instrument scale value data set;
constructing a key point detection network model of the pointer instrument: training a pointer instrument key point detection network model by using the marked pointer instrument key point data set;
s3, using the trained pointer instrument scale value detection network model to predict the scale value of the pointer instrument image to be recognized;
carrying out key point prediction on a pointer instrument image to be recognized by utilizing a trained pointer instrument key point detection network model;
and S4, performing post-processing analysis according to the prediction result of the step S3, and acquiring the current pointer instrument reading in the pointer instrument diagram to be recognized.
2. The method for identifying the reading of the pointer instrument as recited in claim 1, wherein the constructing the image data set of the pointer instrument in the step S1 specifically includes:
s11, shooting a pointer instrument image by using a camera;
s12, acquiring videos shot by each camera, and collecting video streams at different time periods;
s13, processing the collected video stream by adopting a frame extraction method to obtain a plurality of pictures;
s14, summarizing all collected pictures, and screening all the pictures containing pointer instrument images to produce an unlabelled pointer instrument image data set;
s15, marking pointer type instrument image data set:
labeling dial scale values in pictures of the image data set of the unmarked pointer instrument by using an open source labeling tool Labelimg, wherein the labeling type is the dial scale value, and forming a labeled pointer instrument scale value data set;
and labeling dial scales, pointer heads and pointer tails in each picture of the image data set of the unmarked pointer instrument by using an open source labeling tool Labelme to form a labeled pointer instrument key point data set.
3. The pointer instrument reading identification method according to claim 1, wherein in step S2, the pointer instrument scale value detection network model comprises: adding an ASFF self-adaptive feature fusion module in a yolov5 target detection algorithm, and performing weighted fusion on an upper output feature layer;
the ASFF adaptive feature fusion module adopts a formula:
input features of upper layer
Multiplying the weight parameters respectively
Obtaining the feature fusion map of the next layer
Weight parameter
After dimension reduction, the ranges of the two are all [0,1 ] through a softmax function]Internal and the sum is 1.
4. The pointer instrument reading identification method of claim 3, wherein in step S2, the pointer instrument key point detection network model: in step S15, the dial scale, the pointer needle head, and the pointer needle tail marked by the label are respectively a pointer instrument scale feature layer, a pointer instrument needle head feature layer, and a pointer instrument needle tail feature layer;
for a pointer instrument scale feature layer, extracting a pointer instrument scale key point by using an associated Embedding post-processing method; and for the needle head characteristic layer of the pointer instrument and the needle tail characteristic layer of the pointer instrument, extracting the position of the maximum characteristic value in the characteristic diagram as the key point of the needle head of the pointer instrument and the key point of the needle tail of the pointer instrument.
5. The method for identifying reading of pointer instrument as claimed in claim 1, wherein in step S3, the predicting the scale value of the pointer instrument image to be identified by using the trained pointer instrument scale value detection network model specifically comprises:
inputting the pointer instrument image to be identified into a trained pointer instrument scale value detection network model for feature extraction, and performing self-adaptive feature fusion on different output layers according to the extracted features:
and detecting the position and the type of the scale value on the dial plate in the pointer instrument image to obtain the position and the type information of the scale value on the dial plate in the pointer instrument image.
6. The method for identifying the reading of the pointer instrument as recited in claim 1, wherein in step S3, the performing the keypoint prediction on the pointer instrument image to be identified by using the trained pointer instrument keypoint detection network model specifically includes:
inputting the pointer instrument image to be identified into a trained pointer instrument key point detection network model for feature extraction, and directly obtaining the key point information through associated Embedding post-processing according to the extracted features:
and detecting the positions and the types of the dial scales, the pointer needle heads and the pointer tails in the pointer instrument image to obtain the position and the type information of the dial scales, the pointer needle heads and the pointer tails in the pointer instrument image.
7. The pointer instrument reading identification method of claim 1, wherein the specific step of step S4 includes:
s41, aiming at each scale value prediction result of the scale value detection model of the pointer instrument, the positions of the scale values are arranged clockwise and judged according to the principle of from small to large:
if the position of the scale value follows the principle, the prediction result of the scale value is correct;
if the position of the scale value does not conform to the principle, the scale value prediction result is wrong, and the scale value prediction result is rejected;
then traversing each prediction result of the pointer instrument key point detection model, and respectively calculating the position of the scale value and the distance of each dial scale key point: taking the key point with the minimum distance to correspond to the position of the scale value, namely the scale value represented by the key point;
s42, after all the scale values are in one-to-one correspondence with the key points, traversing all the key points according to the clockwise direction:
if there is a key point without corresponding scale value, but the left and right key points have scale values, the scale value of the left key point is set as S n-1 The scale value of the right key point is S n+1 Then the scale value S of the key point n ＝(S n-1 +S n+1 )/2；
After one-time traversal, if the key points which do not correspond to the scale values exist, deleting the key point information;
s43, traversing all the remaining key points with corresponding scale values, calculating the minimum distance from the key point position to the needle head and the needle tail line segment of the pointer, and finding out the key points on two sides of the pointer:
let Sd be the minimum distance from the key point on the left side of the pointer to the pointer n-1 Scale value of S n-1 (ii) a The minimum distance from the key point on the right side of the pointer to the pointer is Sd n+1 Scale value of S n+1 (ii) a The reading of the current pointer instrument is Sn ═ S (S) n+1 +Sd n+1 *S n-1 /Sd n-1 )/(1+Sd n+1 /Sd n-1 ) And obtaining the reading result of the current pointer meter.
8. A system for loading the recognition method of any one of claims 1 to 7, comprising:
the device comprises an image acquisition module, a prediction module loaded with a pointer instrument scale value detection network model and a pointer instrument key point detection network model, and a post-processing analysis module to obtain the current pointer instrument reading in a pointer instrument graph to be identified;
the image acquisition module is loaded with the method of the step S1;
the prediction module loaded with the pointer instrument scale value detection network model and the pointer instrument key point detection network model is used for processing according to the methods of the steps S2 and S3;
the post-processing analysis module is loaded with the method of step S4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210706467.2A CN115019294A (en) | 2022-06-21 | 2022-06-21 | Pointer instrument reading identification method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210706467.2A CN115019294A (en) | 2022-06-21 | 2022-06-21 | Pointer instrument reading identification method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115019294A true CN115019294A (en) | 2022-09-06 |
Family
ID=83077633
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210706467.2A Pending CN115019294A (en) | 2022-06-21 | 2022-06-21 | Pointer instrument reading identification method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115019294A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115457136A (en) * | 2022-11-09 | 2022-12-09 | 杭州远鉴信息科技有限公司 | GIS instrument sulfur hexafluoride data monitoring method and system based on edge calculation |
| CN115880683A (en) * | 2023-03-02 | 2023-03-31 | 江西省水利科学院(江西省大坝安全管理中心、江西省水资源管理中心) | Urban waterlogging ponding intelligent water level detection method based on deep learning |
-
2022
- 2022-06-21 CN CN202210706467.2A patent/CN115019294A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115457136A (en) * | 2022-11-09 | 2022-12-09 | 杭州远鉴信息科技有限公司 | GIS instrument sulfur hexafluoride data monitoring method and system based on edge calculation |
| US11790516B1 (en) * | 2022-11-09 | 2023-10-17 | Hangzhou Yuanjian Information Technology Co., Ltd | Method and system for monitoring GIS instrument sulfur hexafluoride data based on edge computing |
| CN115880683A (en) * | 2023-03-02 | 2023-03-31 | 江西省水利科学院(江西省大坝安全管理中心、江西省水资源管理中心) | Urban waterlogging ponding intelligent water level detection method based on deep learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110059694B (en) | Intelligent identification method for character data in complex scene of power industry | |
| CN106529537B (en) | A kind of digital instrument reading image-recognizing method | |
| CN106529559A (en) | Pointer-type circular multi-dashboard real-time reading identification method | |
| CN108564085B (en) | Method for automatically reading of pointer type instrument | |
| CN109583324A (en) | A kind of pointer meters reading automatic identifying method based on the more box detectors of single-point | |
| CN110992349A (en) | Underground pipeline abnormity automatic positioning and identification method based on deep learning | |
| CN115019294A (en) | Pointer instrument reading identification method and system | |
| CN108830332A (en) | A kind of vision vehicle checking method and system | |
| CN110598693A (en) | Ship plate identification method based on fast-RCNN | |
| CN108764134A (en) | A kind of automatic positioning of polymorphic type instrument and recognition methods suitable for crusing robot | |
| CN111860277B (en) | Safety warning method for airspeed tube sleeve of civil aircraft based on color histogram feature | |
| CN111368690A (en) | Deep learning-based video image ship detection method and system under influence of sea waves | |
| CN114694178A (en) | Method and system for monitoring safety helmet in power operation based on fast-RCNN algorithm | |
| CN114241469A (en) | Information identification method and device for electricity meter rotation process | |
| CN110728269A (en) | High-speed rail contact net support pole number plate identification method | |
| CN116363655A (en) | Financial bill identification method and system | |
| CN108734158B (en) | Real-time train number identification method and device | |
| CN115631488A (en) | Jetson Nano-based fruit maturity nondestructive testing method and system | |
| CN112364844B (en) | Data acquisition method and system based on computer vision technology | |
| CN115311447A (en) | Pointer instrument indicating number identification method based on deep convolutional neural network | |
| CN114662594A (en) | Target feature recognition analysis system | |
| CN114399754A (en) | Digital instrument character detection method based on SW-FSSD | |
| CN113269195A (en) | Reading table image character recognition method and device and readable storage medium | |
| Liu et al. | Automatic detection and recognition method of digital instrument representation | |
| CN115953782A (en) | Pointer type instrument automatic detection reading method based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |