CN112598001B - Automatic ship water gauge reading identification method based on multi-model fusion - Google Patents
Automatic ship water gauge reading identification method based on multi-model fusion Download PDFInfo
- Publication number
- CN112598001B CN112598001B CN202110248503.0A CN202110248503A CN112598001B CN 112598001 B CN112598001 B CN 112598001B CN 202110248503 A CN202110248503 A CN 202110248503A CN 112598001 B CN112598001 B CN 112598001B
- Authority
- CN
- China
- Prior art keywords
- water gauge
- scale
- image
- character
- reading
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/24—Aligning, centring, orientation detection or correction of the image
- G06V10/243—Aligning, centring, orientation detection or correction of the image by compensating for image skew or non-uniform image deformations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/02—Recognising information on displays, dials, clocks
Abstract
The invention discloses a ship water gauge reading automatic identification method based on multi-model fusion, which comprises the following steps: (1) carrying out water gauge area detection on the original image; (2) digging a local image of the water gauge from an original image; (3) carrying out image processing on the water gauge local image; (4) correcting the original image to obtain a corrected original image; (5) scratching the corrected local image of the water gauge, and detecting the scale characters to obtain a scale character detection result; (6) correcting the detection result of the scale characters to obtain the scale reading of the water gauge; (7) establishing a scale reading model; (8) carrying out image segmentation on the corrected original image to obtain the position information of the waterline; (9) inputting the draft line position information into a scale reading model to calculate to obtain the scale reading of the draft line of the water gauge; the invention can effectively solve the problem of automatic reading of the current ship water gauge, and greatly improves the working efficiency; and can be reused in other application scenes related to water gauge reading.
Description
Technical Field
The invention relates to a ship water gauge reading automatic identification method based on multi-model fusion, and belongs to the field of ship water gauge scale detection.
Background
With the development of the ship transportation industry, the reading of draft values of six-side water gauges of a ship becomes the most key factor of the accuracy of weighing and metering of the water gauges, and the original manual visual measurement reading method has the problems of large error, over subjectivity and the like; in addition, the inconvenience of environment observation increases the cost and workload of manual visual measurement, and also increases the risk of personnel safety; in addition to the traditional manual visual inspection method, various solutions have been tried to solve the problem; physical sensor-based methods include: detecting the draught depth of a ship board by installing a double pressure sensor; or ultrasonic waves are transmitted by the ultrasonic sensor, and the position information of the waterline is calculated according to the time required for the ultrasonic waves to be reflected from the water surface to the deck; or directly obtaining the water surface position by using a laser ranging sensor; the methods have the problems of difficult installation, difficult construction, easy environmental influence on precision and the like; therefore, a method and application for automatically reading the water gauge reading based on an image video algorithm are necessary.
The image recognition method comprises three main modules: positioning a waterline, recognizing characters and pre-estimating numerical values; the method based on traditional image recognition comprises the following steps: inputting a water gauge image, detecting the position of a water surface line through an edge processed by the image, locating a scale digital position by utilizing image binaryzation, identifying through methods such as template matching and the like, and finally fitting to estimate a scale value of draught of the water gauge; the method has poor generalization and robustness, is limited to partial water gauge scenes, is easily influenced by factors such as ship rust, water boundary lines and the like, and cannot ensure the water surface line detection positioning and digital identification precision; in recent years, a related method based on deep learning is also provided, scale character recognition and positioning are realized by utilizing a training deep learning network, waterline position information is positioned by utilizing an image segmentation network, and the waterline is estimated by utilizing methods such as a least square method and the like; the method has the problems that the error recognition rate of character positioning and recognition error detection of the whole image is high, and the bounding box of the character positioning is not attached enough, so that a large reading error is caused; in addition, the problems of inclination, rotation angle and the like of the water gauge in the image cannot be solved well; in addition, as the ship body has a rotation inclination angle in a non-plane, the numerical value of the scale characters and the interval between the numerical values present a non-linear state, and the reading error is easy to increase.
Disclosure of Invention
The invention aims to carry out targeted optimization and solution on the problems and the defects of the existing image identification method, so that the invention provides the automatic identification method of the ship water gauge reading based on multi-model fusion, and the method has the characteristics of high flexibility, high accuracy, high generalization robustness and the like; the problem of automatic reading of the current ship water gauge can be effectively solved, and the working efficiency is greatly improved; and can be reused in other application scenes related to water gauge reading.
In order to achieve the above object, the present invention adopts the following technical solutions:
a ship water gauge reading automatic identification method based on multi-model fusion comprises the following steps:
(1) carrying out water gauge area detection on the acquired original image;
(2) digging a local image of the water gauge from an original image;
(3) carrying out image processing on the local image of the water gauge to obtain a changed rotation and translation matrix M;
(4) correcting the original image by the rotation and translation matrix M obtained in the step (3) to obtain a corrected original image;
(5) the corrected water gauge local image is scratched according to the corrected original image rotation center position information and the length and width numerical value, and scale character detection is carried out on the corrected water gauge local image to obtain a scale character detection result, wherein the scale character detection result comprises reading information of scale characters and a boundary surrounding frame;
(6) correcting the detection result of the scale characters to obtain the scale reading of the water gauge;
(7) performing polynomial fitting according to each scale reading in the water gauge scale readings obtained in the step (6) and the corresponding position information thereof to obtain a nonlinear relation between the water gauge scale reading and the position of the longitudinal axis, and establishing a scale reading model;
(8) inputting the corrected original image obtained in the step (4) into a U-net neural network for image segmentation to obtain the position information of the waterline;
(9) and (4) inputting the draft line position information into the scale reading model established in the step (7) to calculate and obtain the scale reading of the draft line of the original image water gauge.
As a further preferred method of the present invention, the specific method of step (1) and step (2) is:
and detecting a water gauge target by using a YOLOv4 water gauge target detection model, wherein the detected target is a water gauge, and extracting a local image of the water gauge from the original image according to the water gauge target detection result information.
As a further preferred aspect of the present invention, a method for training the YOLOv4 water gauge target detection model is: the original image is labeled and patrolled by no less than 5000 pieces manually, the Yolov4 neural network structure takes Darknet53 as a main structure, SPP and PANet structures are introduced to enhance feature extraction and multi-scale prediction, and 3-channel image data with 608 × 608 resolution is input by the network; in the training process, 20% of samples are randomly extracted as a verification set, 80% of samples are taken as a training set, a training platform is dark net, and the training is not less than 200 epochs; after the training is finished, generating a corresponding deep learning water gauge target detection model, testing an untrained original image by using the obtained deep learning water gauge target detection model to obtain a test result, and optimizing and adjusting the deep learning water gauge target detection model according to the test result to finally obtain the YOLOv4 water gauge target detection model.
As a further preferable mode of the present invention, in the step (3), an image processing algorithm of edge detection, binarization, swelling corrosion is applied to the water gauge local image to obtain an external rectangle of the scale character region, and the rotation and translation matrix M for the inclination correction of the water gauge local image is calculated according to the inclination angle and the rotation center of the external rectangle of the scale character region.
As a further preferred method of the present invention, the specific method of detecting the scale characters in the corrected partial image of the water gauge in step (5) is:
and detecting a character target by using a YOLOv4 character target detection model, wherein the character target is a number, a letter M and a decimal point from 0 to 9, 12 types of targets are counted, and a scale character detection result is obtained according to character target detection result information.
As a further preferred aspect of the present invention, a training method of the YOLOv4 character target detection model is: the number of the water gauge local images after artificial marking and correction is not less than 5000, the YOLOv4 neural network structure takes MobileNet3 as a main structure, SPP and PANet structures are introduced to enhance feature extraction and multi-scale prediction, and 3-channel image data with 416 × 416 resolution is input by the network; in the training process, randomly extracting 10% as a test set, 15% as a verification set and 75% as a training set; the training platform is a pyrorch, iteration is carried out for more than 100 thousands of times, after the training is finished, a corresponding deep learning character target detection model is generated, the obtained deep learning character target detection model is used for testing the untrained corrected water gauge local image to obtain a test result, the deep learning character target detection model is optimized and adjusted according to the test result, and finally the YOLOv4 character target detection model is obtained.
As a further preferred aspect of the present invention, the correcting the scale character detection result in step (6) includes:
firstly, correcting a boundary bounding box of each line of scale characters by combining result information of edge detection, then combining detection results of the scale characters of each line, and simultaneously combining identification results of the scale characters of each line;
and secondly, correcting the recognition result of each line of scale characters, and eliminating abnormal data and repairing error data by using the priori knowledge of numerical value sorting of the water gauge scales from top to bottom.
As a further preferred aspect of the present invention, the specific steps of the first step are as follows:
comparing the boundary enclosing frame of the scale characters with the result information of the edge detection; if the boundary of the edge detection is inside the boundary bounding box of the scale character, updating the boundary bounding box result of the scale character according to the boundary of the edge detection to obtain a final boundary bounding box of the scale character; otherwise, not processing, and directly using the boundary bounding box of the scale character as a final scale character boundary bounding box; thereby obtaining a boundary bounding box which is more fit with the scale characters; after the boundary bounding box that each scale character is more laminated is obtained, the scale character detection results of each line are combined, and the scale characters of the same line are ensured to share one large boundary bounding box.
As a further preferable mode of the present invention, in the image segmentation in step (8), the hull region in the image is segmented into the foreground, the water body region is segmented into the background, and the segmentation line between the foreground and the background is the waterline position information of the water gauge.
As a further preferable aspect of the present invention, the scale reading of the draft line of the ship water gauge in the plurality of images photographed frame by frame is calculated, the scale reading result of the draft line of the ship water gauge in each frame of image is drawn, and the average value of the plurality of scale reading results is calculated according to the current patrol water level variation diagram, so as to obtain the scale reading average value result of the draft line of the ship water gauge.
The invention has the advantages that:
according to the invention, the stability and accuracy of each module of the water gauge reading method are enhanced by means of multi-model fusion, the implementation scheme of each module is based on a deep learning technology, a deep learning neural network has the capability of representing image detail characteristics, the multi-model fusion strategy improves the accuracy and the recognition success rate of character positioning, the image segmentation model enables the detection of the position of the waterline to be more accurate, and the environmental interference and the error of reading calculation are reduced; the invention can effectively solve the problems of the automatic reading of the current ship water gauge scale; the method has the characteristics of high flexibility, high accuracy, high generalization robustness and the like, and greatly improves the working efficiency; the automatic identification method for the ship water gauge reading provided by the invention has the advantages of strong practicability, high detection precision and reliable stability, so that the objectivity and accuracy of the ship water gauge draft line scale reading can be ensured, the defect of the conventional ship water gauge draft line scale reading inspection can be overcome, and the weighing and metering requirements of the ship industry are guaranteed; but also can be reused in other application scenes related to water gauge reading.
Drawings
FIG. 1 is a schematic representation of the process steps of the present invention;
FIG. 2 is a schematic view of a local image of a water gauge for area detection matting of the water gauge of the present invention;
FIG. 3 is a schematic view of the partial image correction of the water gauge of the present invention;
FIG. 4 is a schematic diagram of the calibration character detection and calibration character detection result correction of the present invention;
FIG. 5 is a schematic of a non-linear fit of the scale reading model of the present invention;
FIG. 6 is a schematic of the present invention determining draft line position and calculating draft line scale readings.
Detailed Description
The invention is described in detail below with reference to the figures and the embodiments.
As shown in fig. 1, the method for automatically identifying the reading of the water gauge of the ship based on the multi-model fusion provided by the invention comprises the following steps:
(1) carrying out water gauge area detection on the acquired original image;
(2) digging a local image of the water gauge from an original image;
(3) carrying out image processing on the local image of the water gauge to obtain a changed rotation and translation matrix M;
(4) correcting the original image by the rotation and translation matrix M obtained in the step (3) to obtain a corrected original image;
(5) the corrected water gauge local image is scratched according to the corrected original image rotation center position information and the length and width numerical value, and scale character detection is carried out on the corrected water gauge local image to obtain a scale character detection result, wherein the scale character detection result comprises reading information of scale characters and a boundary surrounding frame;
(6) correcting the detection result of the scale characters to obtain the scale reading of the water gauge;
(7) performing polynomial fitting according to each scale reading in the water gauge scale readings obtained in the step (6) and the corresponding position information thereof to obtain a nonlinear relation between the water gauge scale reading and the position of the longitudinal axis, and establishing a scale reading model;
(8) inputting the corrected original image obtained in the step (4) into a U-net neural network for image segmentation to obtain the position information of the waterline;
(9) and (4) inputting the draft line position information into the scale reading model established in the step (7) to calculate and obtain the scale reading of the draft line of the original image water gauge.
The specific method of the step (1) and the step (2) is as follows:
detecting a water gauge target by using a YOLOv4 water gauge target detection model, wherein the detected target is a water gauge, and extracting a local image of the water gauge from an original image according to the water gauge target detection result information; in this embodiment, the method for training the YOLOv4 water gauge target detection model is as follows: the original image is labeled and patrolled by no less than 5000 pieces manually, the Yolov4 neural network structure takes Darknet53 as a main structure, SPP and PANet structures are introduced to enhance feature extraction and multi-scale prediction, and 3-channel image data with 608 × 608 resolution is input by the network; in the training process, 20% of samples are randomly extracted as a verification set, 80% of samples are taken as a training set, a training platform is dark net, and the training is not less than 200 epochs; after the training is finished, generating a corresponding deep learning water gauge target detection model, testing an untrained original image by using the obtained deep learning water gauge target detection model to obtain a test result, and optimizing and adjusting the deep learning water gauge target detection model according to the test result to finally obtain the YOLOv4 water gauge target detection model.
In this embodiment, in step (3), an image processing algorithm of edge detection, binarization, and dilation corrosion is performed on the water gauge local image to obtain an external rectangle of the scale character region, and a rotation and translation matrix M for tilt correction of the water gauge local image is calculated according to an inclination angle and a rotation center of the external rectangle of the scale character region.
The specific method for detecting the scale characters of the corrected local image of the water gauge in the step (5) of the embodiment is as follows:
detecting a character target by using a YOLOv4 character target detection model, wherein the character target is a number, a letter M and a decimal point from 0 to 9, and counting 12 types of targets, and obtaining a scale character detection result according to character target detection result information; in this embodiment, the training method of the YOLOv4 character target detection model is as follows: the number of the water gauge local images after artificial marking and correction is not less than 5000, the YOLOv4 neural network structure takes MobileNet3 as a main structure, SPP and PANet structures are introduced to enhance feature extraction and multi-scale prediction, and 3-channel image data with 416 × 416 resolution is input by the network; in the training process, randomly extracting 10% as a test set, 15% as a verification set and 75% as a training set; the training platform is a pyrorch, iteration is carried out for more than 100 thousands of times, after the training is finished, a corresponding deep learning character target detection model is generated, the obtained deep learning character target detection model is used for testing the untrained corrected water gauge local image to obtain a test result, the deep learning character target detection model is optimized and adjusted according to the test result, and finally the YOLOv4 character target detection model is obtained.
In step (6) of this embodiment, correcting the scale character detection result includes:
firstly, correcting a boundary bounding box of each line of scale characters by combining result information of edge detection, then combining detection results of the scale characters of each line, and simultaneously combining identification results of the scale characters of each line;
secondly, correcting the recognition result of each line of scale characters, and eliminating abnormal data and repairing error data by using the priori knowledge of numerical value sorting of the water gauge scales from top to bottom;
the first step comprises the following specific steps:
comparing the boundary enclosing frame of the scale characters with the result information of the edge detection; if the boundary of the edge detection is inside the boundary bounding box of the scale character, updating the boundary bounding box result of the scale character according to the boundary of the edge detection to obtain a final boundary bounding box of the scale character; otherwise, not processing, and directly using the boundary bounding box of the scale character as a final scale character boundary bounding box; thereby obtaining a boundary bounding box which is more fit with the scale characters; after the boundary bounding box that each scale character is more laminated is obtained, the scale character detection results of each line are combined, and the scale characters of the same line are ensured to share one large boundary bounding box.
In this embodiment, the image segmentation in step (8) is to segment the hull region in the image into a foreground, segment the water body region into a background, and segment a line between the foreground and the background as the waterline position information of the water gauge.
In practical application, the scale reading of the draft line of the ship water gauge in a plurality of images shot frame by frame can be calculated, the scale reading result of the draft line of the ship water gauge in each frame of image is drawn, the average value of the scale reading results is calculated according to the current patrolled and examined water level change diagram, and the average value result of the scale reading of the draft line of the ship water gauge is obtained.
The following is a detailed description of the technical solutions involved in the respective steps:
firstly, the invention carries out water gauge region detection on the whole acquired original image, and aims to obtain local image information of the water gauge and remove the influence on subsequent identification reading caused by shooting or disordered backgrounds of ships; the invention firstly carries out manual labeling on a large number of inspection pictures, trains a Yolov4 water gauge target detection model, and only one type of detection target is provided: a water gauge; according to the water gauge target detection result information, extracting a local water gauge image from the whole original image so as to facilitate subsequent operation; a detailed schematic of this section is shown in fig. 2.
Secondly, the invention carries out image inclination correction on the obtained local image of the water gauge, and aims to rotate the water gauge in a plane to be in a vertical state and reduce reading errors; the method adopts image processing algorithms such as edge detection, binaryzation, swelling corrosion and the like to obtain an external rectangle of a scale character area; calculating a rotation translation matrix M for image tilt correction according to the tilt angle and the rotation center of the external rectangle; the original slightly inclined original image is subjected to M rotation transformation to obtain a positive original image; after obtaining the corrected original image, digging out the corrected local image of the water gauge according to the position information of the rotation center and the length and width values; a detailed schematic of this section is shown in fig. 3.
The method is different from other methods which use OCR to position and recognize the positions of the scale characters, the OCR relates to character direction detection, and the detection recognition rate of the characters of the whole image is low; the invention directly trains an end-to-end target detection network by using a target detection method, and outputs bounding box information and identification information of each character at one time; the Yolov4 neural network structure is still used for training a target detection model, and the detection target is a number, a letter M and a decimal point from 0 to 9, and the total number is 12 types of targets; the invention also designs a character correction module which corrects all detected characters and mainly comprises two aspects of correction, wherein one aspect is that characters in the same row are combined, and then the boundary bounding box of each row of characters is corrected by combining the result information of edge detection so as to obtain the boundary bounding box which is most fit with the scale characters; secondly, correcting the character recognition result of each line, namely correcting the reading information of the scale characters, and eliminating abnormal data and repairing error data by using the priori knowledge sorted from the upper value to the lower value of the water gauge scale to prevent the abnormal data and the repairing error data from interfering with the final reading, wherein the specific schematic diagram of the part is shown in fig. 4; the character recognition result has the phenomenon of recognition error, for example, 19M is recognized as 18M, the prior knowledge of the water gauge scale arrangement is used for correction, the fourth numerical value error can be judged according to the detection result arrangement 22M, 21M, 20M, 18M and 18M, and the character correction module greatly reduces the error rate of the water gauge scale reading.
For the above character recognition task: according to the invention, a deep learning detection method yolov4 is used for detecting a water gauge area firstly, so that the local information of a target is reserved, and more background clutter interference during shooting is removed; after extracting the local image of the water gauge, preprocessing by combining image processing methods such as binaryzation and the like, aiming at correcting the problems of inclination and rotation of the water gauge; detecting the positions and the categories of the scale characters in the corrected water gauge local image by using a deep learning detection method yolov 4; secondly, performing character correction on the character detection result, wherein the correction comprises two aspects, namely extracting edge information for each character position area and further fitting a boundary surrounding frame of the character; secondly, correcting the character recognition result, and correcting the wrong character of the recognition result; the multi-model fusion method is higher in robustness, still has high detection rate for variable environments and different water gauges, and simultaneously solves the problems of image inclination and insufficient character positioning during water gauge shooting by workers.
The invention also establishes a scale reading model; because the ship body is not completely vertical to the plane of the water surface and the parallax problem of the pitch angle also exists due to shooting, the longitudinal axis coordinate (the y-axis coordinate at the lower right corner of the frame surrounded by the character boundary) and the corresponding reading of each scale in the image are in non-linear correlation, or the pixel intervals between the scales are not equidistant; the invention utilizes a nonlinear fitting method to establish a scale reading model, and establishes a two-dimensional correlation diagram according to each scale reading and corresponding position information thereof, wherein the part is specifically shown in figure 5, and a curve is obtained by nonlinear fitting of the invention.
In the invention, a waterline positioning model is also established for obtaining waterline position information, an image segmentation model is trained by utilizing a U-net neural network, the original image after front correction is input into the image segmentation model, segmentation images of a foreground and a background are directly obtained, a segmentation line in the segmentation image is the position of the waterline of the ship water gauge, finally, the waterline position information is input into a scale reading model, and scale reading of the waterline of the original image water gauge is obtained, and the specific schematic diagram of the part is shown in figure 6.
For the waterline positioning task: the invention utilizes the convolution neural network U-net to segment the whole image element, accurately positions the waterline position information, and the U-net can obtain stronger characteristic representation in the image segmentation task to obtain higher segmentation precision, ensure the accuracy of the waterline position information and reduce the reading error.
For the reading value estimation task: the method comprises the steps of establishing a reading model by using a nonlinear fitting method, obtaining a plurality of groups of corresponding scattered points according to the numerical value and longitudinal position information of each scale, and performing polynomial nonlinear fitting to obtain a polynomial relation of all scale positions and readings of the water gauge; directly calculating corresponding scale reading according to the position information of the waterline; the method can solve the reading error caused by the problems that the scale digital part sinks under the water surface, the plane of the ship bends and inclines and the like; in addition, the invention also identifies the multi-frame images and averages the reading results of all scales so as to reduce the influence of surge on the scale reading at a certain moment.
The following describes in detail an implementation process of the automatic identification method for the ship water gauge reading based on multi-model fusion, which is provided by the invention, with reference to a specific embodiment.
Firstly, preparing training data of a water gauge target detection model, manually marking and inspecting not less than 5000 original images, wherein the source of the training data is mainly provided by self-collection and customers; expanding the data volume by off-line data enhancement such as Gaussian noise, rotation, translation, interception and the like; the neural network structure adopts a YOLOv4 target detection model, and the detection targets are only one type of water gauge; the Yolov4 neural network structure takes Darknet53 as a main structure, introduces SPP and PANet structures to enhance feature extraction and multi-scale prediction, and has network input of 3-channel image data with 608 × 608 resolution; in the training process, 20% of samples are randomly extracted as a verification set, 80% of samples are taken as a training set, a training platform is dark net, and the training is not less than 200 epochs; after the training is finished, generating a corresponding deep learning water gauge target detection model, testing an untrained original image by using the obtained deep learning water gauge target detection model to obtain a test result, optimizing and adjusting the deep learning water gauge target detection model according to the test result, and finally obtaining a model with an optimal effect, namely the YOLOv4 water gauge target detection model.
Secondly, preparing training data of a character target detection model; based on the water gauge target detection model training data, the number of the corrected water gauge local images is artificially marked to be not less than 5000, the neural network structure still adopts a YOLOv4 target detection model, and the detection targets are numbers, letters M and decimal points from 0 to 9, which are 12 types in total; because the main content of the water gauge local image is the scale character characteristic, the invention properly reduces the calculated amount of the target detection network in order to reduce the total time consumption when the detection models are cascaded; the YOLOv4 neural network structure takes MobileNet3 as a main structure, SPP and PANet structures are introduced to enhance feature extraction and multi-scale prediction, and the network input is 3-channel image data with 416 × 416 resolution; in the training process, randomly extracting 10% as a test set, 15% as a verification set and 75% as a training set; the training platform is a pyrorch, iteration is carried out for more than 100 thousands of times, after the training is finished, a corresponding deep learning character target detection model is generated, the obtained deep learning character target detection model is used for testing the untrained corrected water gauge local image to obtain a test result, the deep learning character target detection model is optimized and adjusted according to the test result, and finally the model with the optimal effect is obtained, namely the YOLOv4 character target detection model is finally obtained.
Thirdly, preparing an image segmentation model, manually marking and segmenting 2000 training sample images, and taking a ship part as a foreground and a water part as a background; semantic segmentation is carried out by utilizing a deep neural network U-net, the network input resolution is 572 x 572, a training platform is a pytorch, iteration is carried out for more than 50 epochs, after training is completed, a corresponding deep learning semantic segmentation model is generated, an untrained image is tested by utilizing the obtained deep learning target detection model, a test result is obtained, the deep learning target detection model is optimized and adjusted according to the test result, and finally a model with the optimal effect, namely the image segmentation model, is obtained.
At this point, the early preparation work is finished, a detailed process of automatic identification of the ship water gauge reading is shown below, a series of image frames of the ship water gauge scale inspection are input, and one of the image frames is used for example.
Firstly, inputting the image into a water gauge target detection model, adjusting the image scale to 608 x 608, carrying out forward reasoning by a deep neural network, removing overlapped frames by using a non-maximum suppression method, and selecting a target frame result with the maximum response value, namely the water gauge region positioning position, wherein the score threshold value is set to be 0.5, and if the score is lower than the value, the target frame result is directly discarded for carrying out next frame image detection, as shown in fig. 2.
Secondly, intercepting the positioned water gauge area, carrying out edge detection on a local image block to ensure that the image mainly comprises scale characters, removing the interference of background information, and searching for an external rectangle of the whole water gauge area by image processing methods such as expansion corrosion and the like; calculating to obtain a rotation and translation matrix M of the image by using the length, the width and the inclination angle of the external rectangle; obtaining a corrected image after the original input image and the M act, and intercepting a corrected water gauge local image according to the position information and the length and the width of the rotation center in the corrected image, as shown in figure 3; then adjusting the image scale to 416 x 416, removing overlapped frames by using a non-maximum suppression method through forward reasoning of a target detection network, and selecting a target frame result with the maximum response value, namely the position of each scale character; the detection score threshold is set to 0.4, and if the detection score threshold is lower than the value, the detection score threshold is discarded; firstly, carrying out sobel operator filtering on a local image block input into a detection network to obtain an edge detection image, comparing a character detection frame with edge detection information, if an edge detection point exists in the character detection frame, correspondingly shrinking a character detection frame result, and if not, carrying out no processing; after obtaining a detection frame that each character is more close to, merging the character detection results of each line to ensure that the characters of the same line share one large surrounding frame, and merging the character recognition results, for example, 2, 0, and M are merged into 20M; secondly, the recognition result of the character detection is corrected, as shown in fig. 4; wherein, the character detection result has the phenomenon of wrong identification, for example, 19M is identified as 18M, and the invention utilizes the prior knowledge of the scale arrangement of the water gauge to correct; for example, 22M, 21M, 20M, 18M are arranged according to the detection result, the fourth numerical error can be judged, and the fourth numerical error can be modified to 22M, 21M, 20M, 19M, 18M; for another example, between 21M and 20M, characters 8, 4 and 2 exist, and according to the prior knowledge from large to small in vertical arrangement, the second numerical error can be judged and can be modified into 8, 6, 4 and 2.
Thirdly, after boundary box information and recognition result information of all scale characters are obtained, the recognition result is converted into corresponding water gauge scale values, in addition, y-axis image coordinates of the lower right corner of the character boundary box corresponding to each value are obtained, and a scale reading model is established by utilizing the corresponding relation of the two groups of data; the invention utilizes a nonlinear polynomial fitting method to draw a correlation curve between two groups of data, wherein a python mathematical library fitting function is applied to realize the correlation curve, and the specific flow is shown in figure 5; the horizontal axis is the y coordinate value of the lower right corner of each character boundary box, the vertical axis is the scale value of the water gauge corresponding to the character, and the scale readings of all the positions of the y axis can be inquired or calculated according to the curve in the graph.
Then, starting to detect the position information of the waterline of the water gauge; carrying out whole image correction on the frame image, and obtaining a corrected image by utilizing the obtained rotation and translation matrix M and calculation thereof; then, the scale is adjusted to 572 x 572, a U-net image segmentation model is input, the ship body in the image is segmented into a foreground, the water body area is segmented into a background, and the segmentation line of the foreground and the background is the position of the water gauge draught line, as shown in fig. 6; inputting the y coordinate of the draft line position into a scale reading model, and directly calculating to obtain the final scale reading of the draft line of the water gauge; the waterline reading in fig. 6 is 17.78M.
And finally, calculating the scale reading of the ship water gauge waterline in a plurality of images shot frame by frame, drawing the scale reading result of the ship water gauge waterline in each frame of image, calculating the average value of the plurality of scale reading results according to the current patrolled and examined water level change diagram, obtaining the average value result of the scale reading of the ship water gauge waterline, and reducing the influence of surge on the scale reading.
According to the invention, the stability and accuracy of each module of the water gauge reading method are enhanced by means of multi-model fusion, the implementation scheme of each module is based on a deep learning technology, a deep learning neural network has the capability of representing image detail characteristics, the multi-model fusion strategy improves the accuracy and the recognition success rate of character positioning, the image segmentation model enables the detection of the position of the waterline to be more accurate, and the environmental interference and the error of reading calculation are reduced; the invention can effectively solve the problems of the automatic reading of the current ship water gauge scale; the method has the characteristics of high flexibility, high accuracy, high generalization robustness and the like, and greatly improves the working efficiency; the automatic identification method for the ship water gauge reading provided by the invention has the advantages of strong practicability, high detection precision and reliable stability, so that the objectivity and accuracy of the ship water gauge draft line scale reading can be ensured, the defect of the conventional ship water gauge draft line scale reading inspection can be overcome, and the weighing and metering requirements of the ship industry are guaranteed; but also can be reused in other application scenes related to water gauge reading.
In the description herein, reference to the description of the terms "one embodiment," "some embodiments," "an example," "a specific example," "some examples," or "practical application" or the like means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention; in this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example; furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The foregoing shows and describes the general principles, principal features and advantages of the invention; it should be understood by those skilled in the art that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by using equivalent alternatives or equivalent variations fall within the scope of the present invention.
Claims (6)
1. A ship water gauge reading automatic identification method based on multi-model fusion is characterized by comprising the following steps:
(1) carrying out water gauge area detection on the acquired original image;
(2) digging a local image of the water gauge from an original image;
(3) carrying out image processing on the local image of the water gauge to obtain a changed rotation and translation matrix M;
(4) correcting the original image by the rotation and translation matrix M obtained in the step (3) to obtain a corrected original image;
(5) the corrected water gauge local image is scratched according to the corrected original image rotation center position information and the length and width numerical value, and scale character detection is carried out on the corrected water gauge local image to obtain a scale character detection result, wherein the scale character detection result comprises reading information of scale characters and a boundary surrounding frame;
(6) correcting the detection result of the scale characters to obtain the scale reading of the water gauge;
(7) performing polynomial fitting according to each scale reading in the water gauge scale readings obtained in the step (6) and the corresponding position information thereof to obtain a nonlinear relation between the water gauge scale reading and the position of the longitudinal axis, and establishing a scale reading model;
(8) inputting the corrected original image obtained in the step (4) into a U-net neural network for image segmentation to obtain the position information of the waterline;
(9) inputting the draft line position information into the scale reading model established in the step (7) to calculate and obtain the scale reading of the draft line of the original image water gauge;
the method comprises the following steps that (3) an image processing algorithm of edge detection, binarization and expansion corrosion is carried out on a local water gauge image to obtain an external rectangle of a scale character area, and a rotation and translation matrix M for inclination correction of the local water gauge image is calculated according to the inclination angle and the rotation center of the external rectangle of the scale character area;
the specific method for detecting the scale characters of the corrected water gauge local image in the step (5) is as follows:
detecting a character target by using a YOLOv4 character target detection model, wherein the character target is a number, a letter M and a decimal point from 0 to 9, and counting 12 types of targets, and obtaining a scale character detection result according to character target detection result information;
the step (6) of correcting the scale character detection result comprises the following steps:
firstly, correcting a boundary bounding box of each line of scale characters by combining result information of edge detection, then combining detection results of the scale characters of each line, and simultaneously combining identification results of the scale characters of each line;
secondly, correcting the recognition result of each line of scale characters, and eliminating abnormal data and repairing error data by using the priori knowledge of numerical value sorting of the water gauge scales from top to bottom;
the first step comprises the following specific steps:
comparing the boundary enclosing frame of the scale characters with the result information of the edge detection; if the boundary of the edge detection is inside the boundary bounding box of the scale character, updating the boundary bounding box result of the scale character according to the boundary of the edge detection to obtain a final boundary bounding box of the scale character; otherwise, not processing, and directly using the boundary bounding box of the scale character as a final scale character boundary bounding box; thereby obtaining a boundary bounding box which is more fit with the scale characters; after the boundary bounding box that each scale character is more laminated is obtained, the scale character detection results of each line are combined, and the scale characters of the same line are ensured to share one large boundary bounding box.
2. The automatic identification method for the ship water gauge reading based on the multi-model fusion as claimed in claim 1, characterized in that the specific method of the step (1) and the step (2) is as follows:
and detecting a water gauge target by using a YOLOv4 water gauge target detection model, wherein the detected target is a water gauge, and extracting a local image of the water gauge from the original image according to the water gauge target detection result information.
3. The method for automatically identifying the reading of the ship water gauge based on the multi-model fusion as claimed in claim 2, wherein the training method of the YOLOv4 water gauge target detection model is as follows: the original image is labeled and patrolled by no less than 5000 pieces manually, the Yolov4 neural network structure takes Darknet53 as a main structure, SPP and PANet structures are introduced to enhance feature extraction and multi-scale prediction, and 3-channel image data with 608 × 608 resolution is input by the network; in the training process, 20% of samples are randomly extracted as a verification set, 80% of samples are taken as a training set, a training platform is dark net, and the training is not less than 200 epochs; after the training is finished, generating a corresponding deep learning water gauge target detection model, testing an untrained original image by using the obtained deep learning water gauge target detection model to obtain a test result, and optimizing and adjusting the deep learning water gauge target detection model according to the test result to finally obtain the YOLOv4 water gauge target detection model.
4. The method for automatically identifying the reading of the ship water gauge based on the multi-model fusion as claimed in claim 1, wherein the training method of the YOLOv4 character target detection model is as follows: the number of the water gauge local images after artificial marking and correction is not less than 5000, the YOLOv4 neural network structure takes MobileNet3 as a main structure, SPP and PANet structures are introduced to enhance feature extraction and multi-scale prediction, and 3-channel image data with 416 × 416 resolution is input by the network; in the training process, randomly extracting 10% as a test set, 15% as a verification set and 75% as a training set; the training platform is a pyrorch, iteration is carried out for more than 100 thousands of times, after the training is finished, a corresponding deep learning character target detection model is generated, the obtained deep learning character target detection model is used for testing the untrained corrected water gauge local image to obtain a test result, the deep learning character target detection model is optimized and adjusted according to the test result, and finally the YOLOv4 character target detection model is obtained.
5. The automatic identification method for the ship water gauge reading based on the multi-model fusion as claimed in claim 1, wherein in the step (8), the image segmentation is to segment the hull region in the image into the foreground, the water body region into the background, and the segmentation line between the foreground and the background is the waterline position information of the water gauge.
6. The automatic identification method for the ship water gauge reading based on the multi-model fusion as claimed in claim 1, characterized in that the scale reading of the ship water gauge draft line in the multiple images shot frame by frame is calculated, the scale reading result of the ship water gauge draft line in each frame of image is drawn, the average value of the multiple scale reading results is calculated according to the current patrolled water level change diagram, and the scale reading average value result of the ship water gauge draft line is obtained.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110248503.0A CN112598001B (en) | 2021-03-08 | 2021-03-08 | Automatic ship water gauge reading identification method based on multi-model fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110248503.0A CN112598001B (en) | 2021-03-08 | 2021-03-08 | Automatic ship water gauge reading identification method based on multi-model fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112598001A CN112598001A (en) | 2021-04-02 |
| CN112598001B true CN112598001B (en) | 2021-06-25 |
Family
ID=75210279
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110248503.0A Active CN112598001B (en) | 2021-03-08 | 2021-03-08 | Automatic ship water gauge reading identification method based on multi-model fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112598001B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113221898B (en) * | 2021-04-16 | 2022-02-15 | 北京科技大学 | Automatic water level gauge reading method |
| CN113435442B (en) * | 2021-05-07 | 2022-11-25 | 三峡大学 | Water level measuring method and device, water gauge and electronic equipment |
| CN113191348B (en) * | 2021-05-31 | 2023-02-03 | 山东新一代信息产业技术研究院有限公司 | Template-based text structured extraction method and tool |
| CN113378836A (en) * | 2021-06-28 | 2021-09-10 | 北京百度网讯科技有限公司 | Image recognition method, apparatus, device, medium, and program product |
| CN113971779B (en) * | 2021-10-29 | 2022-07-01 | 中国水利水电科学研究院 | Water gauge automatic reading method based on deep learning |
| CN115909298A (en) * | 2022-09-26 | 2023-04-04 | 杭州数聚链科技有限公司 | Cargo ship water gauge scale reading method based on machine vision |
| CN115546793B (en) * | 2022-12-05 | 2023-04-18 | 安徽大学 | Automatic reading method and system for water gauge scales and electronic equipment |
| CN116385984B (en) * | 2023-06-05 | 2023-09-01 | 武汉理工大学 | Automatic detection method and device for ship draft |
| CN116453104B (en) * | 2023-06-15 | 2023-09-08 | 安徽容知日新科技股份有限公司 | Liquid level identification method, liquid level identification device, electronic equipment and computer readable storage medium |
| CN117197048B (en) * | 2023-08-15 | 2024-03-08 | 力鸿检验集团有限公司 | Ship water gauge reading detection method, device and equipment |
| CN116824570B (en) * | 2023-08-30 | 2023-11-24 | 江苏省泰州引江河管理处 | Draught detection method based on deep learning |
| CN117315664B (en) * | 2023-09-18 | 2024-04-02 | 山东博昂信息科技有限公司 | Scrap steel bucket number identification method based on image sequence |
| CN117671510A (en) * | 2024-02-02 | 2024-03-08 | 西安热工研究院有限公司 | Ship load capacity calculation method based on target detection and semantic segmentation algorithm |
| CN117788463A (en) * | 2024-02-26 | 2024-03-29 | 中邮建技术有限公司 | Ship draft detection method based on video AI and multi-mode data fusion |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102295061A (en) * | 2011-05-19 | 2011-12-28 | 上海海事大学 | Automatic detection method of ship water gauge based on image processing |
| CN102975826A (en) * | 2012-12-03 | 2013-03-20 | 上海海事大学 | Portable ship water gauge automatic detection and identification method based on machine vision |
| CN107563330A (en) * | 2017-09-04 | 2018-01-09 | 南京邮电大学 | A kind of horizontal tilt car plate antidote in monitor video |
| CN110276285A (en) * | 2019-06-13 | 2019-09-24 | 浙江工业大学 | A kind of shipping depth gauge intelligent identification Method in uncontrolled scene video |
| CN110427933A (en) * | 2019-06-20 | 2019-11-08 | 浙江大学 | A kind of water gauge recognition methods based on deep learning |
| CN111652213A (en) * | 2020-05-24 | 2020-09-11 | 浙江理工大学 | Ship water gauge reading identification method based on deep learning |
| CN112001339A (en) * | 2020-08-27 | 2020-11-27 | 杭州电子科技大学 | Pedestrian social distance real-time monitoring method based on YOLO v4 |
| CN112215074A (en) * | 2020-09-10 | 2021-01-12 | 鲁东大学 | Real-time target identification and detection tracking system and method based on unmanned aerial vehicle vision |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108549894B (en) * | 2018-04-03 | 2021-08-10 | 哈尔滨哈船智控科技有限责任公司 | Automatic reading method for ship water gauge image based on deep learning algorithm |
| CN108921165B (en) * | 2018-06-21 | 2022-04-22 | 江苏南水水务科技有限公司 | Water level identification method based on water gauge image |
| CN110472636B (en) * | 2019-07-26 | 2022-10-14 | 四创科技有限公司 | Deep learning-based water gauge E-shaped scale identification method |
-
2021
- 2021-03-08 CN CN202110248503.0A patent/CN112598001B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102295061A (en) * | 2011-05-19 | 2011-12-28 | 上海海事大学 | Automatic detection method of ship water gauge based on image processing |
| CN102975826A (en) * | 2012-12-03 | 2013-03-20 | 上海海事大学 | Portable ship water gauge automatic detection and identification method based on machine vision |
| CN107563330A (en) * | 2017-09-04 | 2018-01-09 | 南京邮电大学 | A kind of horizontal tilt car plate antidote in monitor video |
| CN110276285A (en) * | 2019-06-13 | 2019-09-24 | 浙江工业大学 | A kind of shipping depth gauge intelligent identification Method in uncontrolled scene video |
| CN110427933A (en) * | 2019-06-20 | 2019-11-08 | 浙江大学 | A kind of water gauge recognition methods based on deep learning |
| CN111652213A (en) * | 2020-05-24 | 2020-09-11 | 浙江理工大学 | Ship water gauge reading identification method based on deep learning |
| CN112001339A (en) * | 2020-08-27 | 2020-11-27 | 杭州电子科技大学 | Pedestrian social distance real-time monitoring method based on YOLO v4 |
| CN112215074A (en) * | 2020-09-10 | 2021-01-12 | 鲁东大学 | Real-time target identification and detection tracking system and method based on unmanned aerial vehicle vision |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112598001A (en) | 2021-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112598001B (en) | Automatic ship water gauge reading identification method based on multi-model fusion | |
| CN107543828B (en) | Workpiece surface defect detection method and system | |
| CN111862064B (en) | Silver wire surface flaw identification method based on deep learning | |
| CN113160192B (en) | Visual sense-based snow pressing vehicle appearance defect detection method and device under complex background | |
| CN108776140B (en) | Machine vision-based printed matter flaw detection method and system | |
| CN110148130B (en) | Method and device for detecting part defects | |
| CN108764229B (en) | Water gauge image automatic identification method based on computer vision technology | |
| CN110853015A (en) | Aluminum profile defect detection method based on improved Faster-RCNN | |
| CN110992349A (en) | Underground pipeline abnormity automatic positioning and identification method based on deep learning | |
| CN110097547B (en) | Automatic detection method for welding seam negative film counterfeiting based on deep learning | |
| CN107230203A (en) | Casting defect recognition methods based on human eye vision attention mechanism | |
| CN111652213A (en) | Ship water gauge reading identification method based on deep learning | |
| CN112037203A (en) | Side surface defect detection method and system based on complex workpiece outer contour registration | |
| CN114693610A (en) | Welding seam surface defect detection method, equipment and medium based on machine vision | |
| CN113689429B (en) | Wood board defect detection method based on computer vision | |
| CN108711148A (en) | A kind of wheel tyre defect intelligent detecting method based on deep learning | |
| CN111415339B (en) | Image defect detection method for complex texture industrial product | |
| CN114495098B (en) | Diaxing algae cell statistical method and system based on microscope image | |
| CN114266764A (en) | Character integrity detection method and device for printed label | |
| CN111222507A (en) | Automatic identification method of digital meter reading and computer readable storage medium | |
| CN110728269B (en) | High-speed rail contact net support pole number plate identification method based on C2 detection data | |
| CN113705564B (en) | Pointer type instrument identification reading method | |
| CN111178405A (en) | Similar object identification method fusing multiple neural networks | |
| CN114004858A (en) | Method and device for identifying aviation cable surface code based on machine vision | |
| CN113962929A (en) | Photovoltaic cell assembly defect detection method and system and photovoltaic cell assembly production line |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |