CN107229930B

CN107229930B - Intelligent identification method for numerical value of pointer instrument

Info

Publication number: CN107229930B
Application number: CN201710297198.8A
Authority: CN
Inventors: 马波; 蔡伟东; 江志农
Original assignee: Beijing University of Chemical Technology
Current assignee: Beijing University of Chemical Technology
Priority date: 2017-04-28
Filing date: 2017-04-28
Publication date: 2021-01-19
Anticipated expiration: 2037-04-28
Also published as: CN107229930A

Abstract

The invention belongs to the field of machine vision, and relates to a pointer instrument numerical value intelligent identification method and device. The invention provides a recognition method and a recognition device which adopt a convolutional neural network model, are based on a model training method of a confrontation sample, intelligently recognize the numerical value of a pointer instrument by the training model and have a camera shooting angle self-adaptive adjustment technology. The device possesses shooting angle self-adaptation function, can carry out the self-correction to shooting angle, need not to carry out accurate adjustment to it before the use. According to the method, the confrontation sample is adopted to train the convolutional neural network recognition model, the workload of original data acquisition is greatly reduced, multiple instrument panel numerical values can be recognized simultaneously, multiple cameras do not need to be adopted for shooting independently, and the cost is saved. The invention has the characteristics of high robustness, high accuracy, high identification speed, strong operability, convenient transplantation and the like.

Description

Intelligent identification method for numerical value of pointer instrument

Technical Field

The invention belongs to the field of machine vision, relates to a pointer instrument numerical value intelligent identification method and device, and particularly relates to an identification method and device which adopts a confrontation sample to train a convolutional neural network identification model and can perform self-correction.

Background

The pointer instrument is stable and reliable, is convenient to maintain, can reflect the measured change trend, and has wide application in the industries of chemical industry, electric power, automobiles and the like. However, most of the existing pointer instruments are recognized by human eyes, which not only wastes time and labor but also has low efficiency, and the reading is greatly influenced by subjective factors, so that it is necessary to develop a set of automatic recognition device for the numerical value of the pointer instrument.

At present, most of pointer instruments are identified by preprocessing images through image processing means such as binarization, morphological transformation, skeleton extraction and the like, then instrument pointers are extracted through Hough transformation, and instrument readings are calculated through pointer deflection angles. The method has a plurality of problems, such as interference of other characteristics (anti-seismic oil lines and the like) on the dial plate on straight line extraction of the pointer, determination of pointer steering, correction of inclined images, image pollution and the like; meanwhile, aiming at the automatic identification of the numerical value of the pointer instrument, people fix the camera to ensure the shooting angle of the camera, but in practical application, some equipment systems work intermittently, and the relative position of the instrument and the camera changes, for example, a nuclear power emergency diesel unit can be started for use only in an emergency state and a periodic test, the parameter measurement of the pointer instrument of the equipment is not suitable for adopting the fixed camera, on one hand, the identification device occupies space, most of time is in an idle state, and on the other hand, the relative position of the camera and the instrument changes due to the reasons of equipment maintenance and the like in the using process; meanwhile, the automatic identification methods for the numerical values of the pointer type meters aim at the identification of a single meter, but in practical application, a plurality of parameters of the pointer type meters are required to be read, and the pointer type meters are of the same type and different types.

Chinese patent No. CN 102521560B entitled "high robust instrument pointer image recognition method" proposes a method for recognizing the numerical value of a pointer instrument by using Hough transform and center projection method, which requires manual adjustment of the position of a camera to align the camera with an instrument panel, and during actual use, the relative position between the camera and the instrument changes due to the maintenance of equipment, and the camera and the instrument cannot be extracted to the instrument position for recognition, so that the operability of the method is poor, and meanwhile, the method requires a mouse to select the pointer center area and the dial outer contour, and is not suitable for use in scenes requiring real-time continuous recognition.

Chinese patent No. 201410855634.5, entitled "reading identification method and apparatus for pointer instrument", describes a method for automatically identifying the numerical value of a pointer instrument by using a convolutional neural network, which divides a dial to be identified into a plurality of parts, outputs the probability that a pointer is located in different parts by using the convolutional neural network, and obtains the numerical value of the pointer according to the position of the maximum probability part in the dial. According to the method, all data are original data in the process of training the model, on one hand, the collection workload is large, and on the other hand, the generalization capability of the model is not strong; meanwhile, the method requires that the division starting position and the division direction of the dial plate image to be identified and the sample dial plate image need to be consistent when the images are divided, which limits the practical use of the method.

Disclosure of Invention

Aiming at the defects of the existing method, the invention provides a model training method based on a confrontation sample by adopting a convolutional neural network model, and the recognition method and the device thereof have the advantages that the training model carries out intelligent recognition on the numerical value of a pointer instrument and have a camera shooting angle self-adaptive adjustment technology. The invention comprises a device and a method for identifying the numerical value of a pointer instrument, which are respectively described below.

The set of identification device is shown in attached figure 1 and comprises a support (1), a program control holder (2), a camera (3), a cross positioning mark (5) and a computer. Firstly, a support (1) is placed in front of a dial (4) to be identified, then a program control tripod head (2) fixedly connected with a camera (3) is fixed on the support (1), a left cross positioning mark and a right cross positioning mark are marked on the surface of the instrument panel, and then the shooting angle and the zoom multiple of the camera (3) are respectively adjusted by the system through the program control tripod head (2) and the camera, so that the shooting preset requirements are met (the specific preset requirements are explained in the specific implementation mode, the same is applied below).

The set of identification device has the characteristics that: the identification device has a shooting angle self-adaption function, can self-correct the shooting angle, judges whether the shooting angle and the zoom multiple of the camera are proper or not by detecting two cross positioning targets on the left and right sides of the surface of the instrument panel, and respectively adjusts the shooting angle and the zoom multiple of the camera by the program-controlled cradle head and the camera when the preset requirement cannot be met so as to meet the shooting requirement; the device is easy to install, compared with other methods, the position of the camera and the position of the instrument are not required to be completely fixed, the camera is only required to be approximately opposite to the instrument panel and can be installed at any time, the camera can be collected when the camera is not required to be used, and the same set of device can be applied to different places.

The method for intelligently identifying the numerical value of the pointer instrument provided by the invention adopts the generated confrontation sample to train the convolutional neural network, and utilizes the model obtained by training to identify the numerical value of the pointer instrument, and the implementation flow of the method is shown as the attached figure 2. The method specifically comprises two parts of model training and model identification.

The model is trained by adopting confrontation samples and original data, the implementation process is shown in the left half part of the attached drawing 2, and the method specifically comprises four parts, namely model construction, sample acquisition, generation of the confrontation samples and model training. The model is constructed by constructing the structure of the whole convolutional neural network, which comprises an input layer, an output layer, a convolutional layer, a pooling layer and a full-connection layer; the acquisition of the sample is to acquire an original image of the dial plate to be identified and prepare data for next generation of the confrontation sample and training of the model; the generation of the countermeasure sample is to perform one or more processing means of affine and perspective transformation, shading change, partial region cutting and noise addition on the acquired original image to obtain a large number of countermeasure samples which can be used for model training; the training of the model is to input the collected original image data and the generated countermeasure sample data into a pre-constructed convolutional neural network recognition model for training, and obtain the model which can be applied to numerical value recognition of the pointer instrument by adjusting the parameters of each node of the model.

The flow of the model identification part is shown in the right half of the attached figure 2, and comprises four parts of self-correction of a camera, image preprocessing, image input of a trained model and model output of an identification result. The self-correction of the camera is that the camera automatically adjusts the shooting angle and the zoom multiple through a self-correction program so as to meet the shooting requirement; the image preprocessing comprises the steps of converting an RGB image to be recognized, which is acquired by a camera, into a gray image, extracting one or more instrument panels in the image and marking the instrument panels; the image input trained model is a convolutional neural network recognition model trained before inputting the extracted one or more instrument panel images; the model output result is the recognition result of the image to be recognized which is output by the trained recognition model after the image to be recognized is input.

Compared with the prior art, the invention has the following characteristics:

(1) the device has the self-adaptive function of the shooting angle, can self-correct the shooting angle, and does not need to precisely adjust the shooting angle before use. The whole set of device is easy to install, so that the stability of the system and the mobility of the device are improved, a set of device can be applied to various scenes, and the operability of the method and the repeated use of the device are greatly improved;

(2) the method adopts the confrontation sample to train the convolutional neural network recognition model, so that the workload of original data acquisition is greatly reduced, the method has good anti-interference capability on light, dial plate pollution, image deflection and the like, and experiments prove that the method has high robustness;

(3) the method can be used for extracting and marking multiple instrument panels, and can be used for identifying the numerical values of the multiple instrument panels at the same time, so that multiple cameras are not required to be used for shooting independently, and the cost is saved;

(4) the method changes the traditional mode of computing by using a CPU (Central Processing Unit) alone, combines the high-speed parallel computing capability of a GPU (graphic Processing Unit) for Processing, greatly improves the identification efficiency, and can be applied to real-time identification.

Compared with the prior art, the method and the device have the characteristics of high robustness, high accuracy, high identification speed, strong operability, convenience in transplantation and the like.

Drawings

FIG. 1 is a diagram of an intelligent identification device for numerical values of a pointer instrument; 1. support 2, program control holder 3, camera 4, dial 5 to be identified cross location mark

FIG. 2 is a flow chart of a method for intelligently identifying numerical values of a pointer instrument;

FIG. 3 is a diagram of a convolutional neural network recognition model;

FIG. 4 is a flow chart of counter sample generation for the pointer instrument;

FIG. 5 is a generated picture of a confrontation sample of the pointer instrument;

(1) original picture (2) pointer-free dashboard separated from original picture

(3) A pointer separated from the original picture; (4) countercheck sample pictures generated by (5) and (6)

FIG. 6 is a flow chart of pointer instrument numerical model identification;

FIG. 7 is a view showing the result of numerical value recognition of the pointer type meter;

FIG. 8 is a diagram showing the result of on-line identification of the numerical value of the pointer instrument.

Detailed Description

The invention aims to provide the method and the device for intelligently identifying the numerical value of the pointer instrument, which have the advantages of strong operability, high identification accuracy, high speed and high robustness.

This recognition device's a big characteristic is for possessing shooting angle self-adaptation function, and the prefabricated cross alignment mark of accessible carries out self-correction, does not need the manual work to adjust. The device is shown in attached figure 1 and comprises a support (1), a program control holder (2), a camera (3), a cross positioning mark (5) and a computer. Firstly, a support (1) is placed in front of a dial (4) to be identified, then a program control tripod head (2) fixedly connected with a camera (3) is fixed on the support (1), a left cross positioning mark and a right cross positioning mark are marked on the surface of the dial, then the camera shoots images to detect the positions of the two cross positioning marks, the deviation degree (expressed by a pixel value) of the upper position, the lower position, the left position and the right position of the cross positioning marks in the images and the initial set position of the system is judged, and then the system respectively adjusts the shooting angle and the zoom multiple of the camera (3) through the program control tripod head (2) and the camera to enable the deviation degree to reach the requirement (the initial set pixel value), namely the preset requirement of shooting is met.

The pointer instrument numerical value intelligent identification method provided by the invention is characterized in that a convolutional neural network is trained by adopting a generated confrontation sample, and the identification is carried out by using a model obtained by training, and specifically comprises two parts of model training and model identification.

One structure of the recognition model is shown in fig. 3 and consists of an input layer, an output layer, 5 convolution pooling layers (1-5 layers) and 2 full-connection layers (6-7 layers). The size of the input layer is 128 × 128, and the input layer corresponds to the size of the gray-scale picture of the instrument panel to be identified; the first layer is a convolution pooling layer, the size of a convolution kernel is 5 × 5, 48 convolution kernels are arranged, 2 × 2 maximum pooling is adopted, the output tensor size of the layer is 62 × 48, 62 × 62 is the size of a picture after convolution, and 48 is the number of the convolution kernels; the 2 nd layer to the 4 th layer are convolution pooling layers, the size of convolution kernels is 3 x 3, the number of the convolution kernels is 128, 192 and 256 respectively, and 2 x 2 maximum pooling is adopted; the 5 th layer is a convolution layer, the size of the convolution kernel is 3 x 3, 338 convolution kernels are arranged, the pooling operation is not carried out, the tensor size output by the convolution pooling layer is 6 x 338, 6 x 6 is the corresponding picture size, and 338 is the number of the convolution kernels; 6. the 7 layers are two full-connection layers, the number of the neurons is 1024 and 512 respectively, the output layer is finally the output layer, the size is 200, the classification number of the picture is corresponded, the numerical value represents the probability value of the input picture to be recognized belonging to each class, the class with the maximum probability value is selected as the recognition result, and then the recognition result is converted into the actual numerical value according to the measuring range. The identification model structure is one of the identification model structures, and the identification model structure in practical application can adjust the layer number and the model parameters of the convolution pooling layer and the full-connection layer according to the type, the range, the identification precision and the like of the instrument panel to be identified to obtain a proper identification model structure.

The generation process of the confrontation sample is shown in the attached figure 4, and comprises the following steps:

step 400, collecting one or more instrument panel original images of the same type as the instrument panel to be identified, separating the pointers of the instrument panels, and generating an instrument panel picture (such as (2) of fig. 5) without the pointers and a separate pointer picture (such as (3) of fig. 5);

step 401, finding the rotation center of the pointer on the dashboard picture without the pointer, and determining the type of the generated sample picture, i.e. the type of the output layer of the identification model, according to the measuring range of the meter and the required identification precision. For example, in the present embodiment, the range is 0-10Mpa, and the recognition accuracy is 0.05Mpa, which is classified into 200 categories;

and step 402, superposing the two pictures according to the rotation center by using the pointer separated in the step 401 and the instrument panel without the pointer, and simultaneously performing processing means such as random affine transformation and perspective transformation, random cutting part area, random brightness change of the whole image, random noise point addition and the like on each picture in the process of generating each picture to obtain 200 types of instrument panel images including label files, wherein the number of the pictures of each type is 60 or more by the method. As shown in fig. 5 (4), (5), and (6), the generated partial confrontation sample picture;

and step 403, generating a data set for training the recognition model by using the original picture, the generated confrontation sample picture and the respective labels thereof.

The training process of the recognition model is to put the picture data set generated in the step 400-403 into the pre-constructed convolutional neural network (80% of data is used for training the model, and 20% of data is used for detecting the accuracy of the model), and by continuously adjusting the parameters of each node in the model, the parameters are enabled to reach the predetermined requirement (the accuracy is 70% or more), and experiments show that the average error of model recognition under the accuracy threshold is within 1%, and the recognition model is successfully trained.

The model identification process is shown in fig. 6, and includes the following steps:

step 600, placing the device in front of the instrument panel to be identified, enabling the camera to approximately face to the instrument position, and fixing a left cross positioning mark and a right cross positioning mark at a preset position on the surface of the instrument panel (the preset position needs to be consistent with the preset position of the cross positioning mark in the self-correcting program);

step 601, starting a system, wherein a camera detects the positions of cross positioning targets in an image, the deviation degree (represented by a pixel value) between the positions of a left cross positioning target and a right cross positioning target in the image and a preset position is enabled to meet the requirement through the horizontal rotation and the vertical rotation of a program control holder and the optical zoom of the camera, and if the self-correction adjustment fails for 50 times and the preset requirement of shooting by the camera cannot be met (the preset requirement is stated in the front), the system self-checks and alarms;

step 602, a camera collects an instrument panel image, graying the collected image, then reducing the dimension, extracting each circular instrument panel in the image through probability Hough circle transformation, marking the circular instrument panel according to the coordinates of each circle, converting the coordinates of the circular instrument panel into the image before dimension reduction, and segmenting the image of each instrument panel with the marks from the image before dimension reduction;

step 603, inputting one or more dashboard images extracted from one or more images in step 602 into the pre-trained convolutional neural network recognition model for calculation. Particularly, when the image acquisition frequency is high (more than 10FPS), the identification model in the step runs on the GPU, and the parallel operation processing can greatly improve the identification speed;

step 604, converting the output result of the recognition model in the step 603 into an actual reading according to the measuring range of the dial plate to be recognized, then storing the actual reading according to the corresponding mark in the step 602, and then performing the next round of recognition.

Recognition effect

The method and the device are adopted for testing, 50 pointer instrument pictures are adopted for testing in the test 1 process, the comparison of results is shown in the attached figure 7, the solid line is a model identification result, the dotted line is a human eye reading value, and the average error is within 1%; in the process of the test 2, the online identification effect is tested, the instrument adopts a pointer type thermometer, the thermometer is placed into hot water from room temperature, then the identification is carried out while the video is shot, the frame rate is 30FPS, the thermometer adopts a bimetallic thermometer, the precision level is 1.5, the result is shown in the attached figure 8, the solid line is the model identification result, and the dotted line is the reading value of human eyes.

Because the thermometer is not high in precision, the situation of blockage can exist, the heat transfer process is slow, the acquisition frequency is high, a step shape is formed on a curve graph, the identification results are basically consistent through comparison of human eyes, and the average error is within 1%.

The present invention is described in detail with reference to the drawings. However, the present invention is not limited to the specific implementation steps and the structure of the apparatus, and any modification or replacement of the related implementation steps based on the above, and any local adjustment of the related implementation steps based on the above are within the spirit of the present invention.

Claims

1. A method for intelligently identifying numerical values of a pointer instrument comprises two parts, namely model training and model identification;

the method is characterized in that: the model is trained by adopting a countermeasure sample and original data, and specifically comprises four parts, namely model construction, sample acquisition, countermeasure sample generation and model training; the model is constructed by constructing the structure of the whole convolutional neural network, which comprises an input layer, an output layer, a convolutional layer, a pooling layer and a full-connection layer; the acquisition of the sample is to acquire an original image of the dial plate to be identified and prepare data for next generation of the confrontation sample and training of the model; the generation of the countermeasure sample is to carry out one or more of random affine and perspective transformation, random brightness change, random partial region cutting and random noise addition on the acquired original image to obtain the countermeasure sample of the model training; the training of the model is to input the collected original image data and the generated countermeasure sample data into a pre-constructed convolutional neural network recognition model for training, and obtain the model which can be applied to the numerical value recognition of the pointer instrument by adjusting the parameters of each node of the model;

the model identification part process comprises four parts, namely self-correction of a camera, image preprocessing, image input of a trained model and model output of an identification result; the self-correction of the camera is that the camera automatically adjusts the shooting angle and the zoom multiple through a self-correction program so as to meet the shooting requirement; the image preprocessing comprises the steps of converting an RGB image to be recognized, which is acquired by a camera, into a gray image, extracting one or more instrument panels in the image and marking the instrument panels; the image input trained model is a convolutional neural network recognition model trained before inputting the extracted one or more instrument panel images; the model output result is that the trained recognition model outputs the recognition result of the image to be recognized after the image to be recognized is input;

the generation of the confrontation sample specifically comprises the following steps:

step 400, collecting one or more instrument panel original images of the same type as the instrument panel to be identified, separating pointers of the instrument panels, and generating an instrument panel picture without the pointers and an independent pointer picture;

step 401, finding the rotation center of the pointer on the instrument panel picture without the pointer, and determining the type of the generated sample picture, namely the type of the output layer of the identification model, according to the measuring range of the instrument and the required identification precision;

step 402, superposing the two pictures according to the rotation center by using the pointer separated in the step 401 and the instrument panel without the pointer, and simultaneously performing random affine transformation and perspective transformation, random cutting part area, random light and shade change of the whole image and random noise point addition on each picture in the process of generating each picture to obtain various instrument panel images including label files, wherein the number of each kind of picture is 60 or more by the method;

step 403, generating a data set for training and identifying the model by using the original picture, the generated confrontation sample picture and respective labels thereof;

the training process of the recognition model is to put the image data set generated in the steps 400-403 into a pre-constructed convolutional neural network, and to continuously adjust the parameters of each node in the model so as to achieve the accuracy of more than 70%.