CN115620014B - Pipeline instrument flow chart information extraction method and equipment based on deep learning - Google Patents
Pipeline instrument flow chart information extraction method and equipment based on deep learning Download PDFInfo
- Publication number
- CN115620014B CN115620014B CN202211552869.8A CN202211552869A CN115620014B CN 115620014 B CN115620014 B CN 115620014B CN 202211552869 A CN202211552869 A CN 202211552869A CN 115620014 B CN115620014 B CN 115620014B
- Authority
- CN
- China
- Prior art keywords
- point
- points
- line
- information
- track
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/28—Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30241—Trajectory
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The invention provides a pipeline instrument flow chart information extraction method and equipment based on deep learning, which belong to the field of target detection in computer vision, wherein PDF or paper pipeline instrument flow charts need to be converted into file types in a CAD format, and a large amount of time is spent on manual redrawing work, while the existing target detection algorithm cannot effectively identify equipment with large size difference on the pipeline instrument flow charts, and simultaneously has a difficult problem on identification of dotted lines, linear lines, mark lines and equipment names.
Description
Technical Field
The invention belongs to the field of target detection in computer vision, and particularly relates to a pipeline instrument flow chart information extraction method and equipment based on deep learning.
Background
When a pipeline instrument flow chart (PID) of a PDF version or a paper version is converted into a file in a CAD format, manual redrawing is needed, time and labor are consumed, and when a target detection method is used for identifying the PID diagram, the accuracy is low when equipment and line shapes with different sizes, such as a solid line, a dotted line, lines with different thicknesses and similar characters, such as '5, S', are processed, and false detection is often caused. At present, the common methods for extracting flow chart information or detecting equipment of the pipeline meter comprise the following steps:
1. deep learning-based PID (proportion integration differentiation) drawing analysis method, computer system and medium
Firstly, establishing an identification model set; traversing the PID drawing image by using a first identification frame to obtain a pipeline identification data set; traversing the PID drawing image by using a second identification frame to obtain a device identification data set; performing data fusion in the pipeline identification data set and the device identification data set by using a non-maximum suppression algorithm to obtain pipeline data and device data; and traversing each pipeline by using a clustering association Attrack algorithm, identifying a device associated with the pipeline, and forming a corresponding relation between the pipeline and the device in the PID drawing. When the method is used for identifying the equipment sizes in the PID graph, the size difference is large, the character similarity identification accuracy is low, and meanwhile, a large amount of manual calibration work is needed.
2. Intelligent PID drawing identification and redrawing method
The method provides a PID drawing intelligent identification and redrawing method, and relates to the field of PID drawing identification, image processing and computer aided design. The invention realizes the positioning, classification, character detection and identification of the drawing in the PID drawing image by adopting the methods of template matching, deep learning, pixel point-based pipeline identification algorithm and merging rule, associates the drawing with the character by using a regular expression and finishes the redrawing of the PID drawing by using PID drawing software, but the method carries out the image matting based on the template, has the problems of low matching rate of equipment and missed detection, and has the problem of higher false detection rate of linear shape by using a residual error network.
3. Second-order identification method for PID drawing picture information
The method discloses a second-order identification method of PID drawing picture information, and relates to the field of image identification image classification, in particular to the identification problem of PID drawing pictures. The method combines the traditional machine learning and deep learning strong supervision to realize accurate identification and classification of the PID drawing pictures, aims at the problem that the intra-class difference in the PID pictures is small, realizes primary classification according to the common basic geometric figure characteristics of the pictures by adopting a normalized correlation coefficient and HOG characteristic extraction correlation calculation mode in the first stage, adds key parts to the pictures with high similarity in the deep learning strong supervision mode in the second stage, improves the performance of the model in classifying and identifying the pictures by improving the perception capability of the model on the intra-class difference parts, but has lower identification accuracy rate on the equipment with large size difference and poorer linear identification effect.
Disclosure of Invention
In view of the above problem, a first aspect of the present invention provides a method for extracting pipeline meter flowchart information based on deep learning, including the following steps:
s1, for the legendD L Acquiring track information and a corresponding label of each device; PID drawing for a whole set of pipeline instrument flow chartD={{D 1 , …, D j ,…, D n }, D L }WhereinD j Is a drawing in the drawing sheet D,nthe number of drawings is shown and indicated,D L is a legend, a legendD L Stipulate the drawingsDWhat each symbol in (1) represents;
s2, from{D 1 , …, D j ,…, D n }Any one of the drawingsD j Performing an iterative process from S3 to S11;
s3, for the drawingD j First, a matrix representation of the drawing is obtainedM' j Performing preprocessing operation to obtain a drawing represented by a matrixM j ；
S4, to the matrixM j Carrying out an iterative process from S5 to S10;
s5, fromM j In which a point other than 0 is randomly selectedp(x 1 , y 1 ) Then, the line width is calculated from the vertical direction and the horizontal direction of the point based on the boundary determination ruledDirection of the sum lineθIf, ifM j If the values are all equal to 0, jumping to S11;
s6, according to step length, along the line where the last point in the drawing is locatedsMove to the next pointq(ii) a Direction of line based on last pointθ，According to the formulax 2 = cell(x 1 + s*cos θ), y 2 = cell(y 1 + s*sin θ)Solving for the coordinates of the next pointq(x 2 , y 2 )，sThe step size for each of the moves is indicated,cellindicating rounding up, setpointpToqIs branch B, judgeqWhether or not it has been walked by other branchesIf q isIf the curve is not in the walking curve, the step S7 is carried out; if it is notqIn the curve that has already been walked, terminate the branch, jump to S9;
s7, finding out points with changed line widths; if it is notM j [x 2 , y 2 ]If the value is greater than 0, solving the point based on the boundary judgment ruleqLine width ofd’Direction of the sum lineθ’If, ifd’Is equal todSetting up a matrixM j MidpointpToqWidth ofdIs 0 while jumping to S6, otherwise the step size is reducedsRecalculatingqUntil founddIs not equal tod’Critical point ofC, entering intoS8; if it is notM j [x 2 , y 2 ]Less than or equal to 0, reducing step lengthsSkipping to S6; if when it is usedsWhen the value is equal to 1, the reaction solution is,M j [x 2 , y 2 ]and alsoLess than or equal to 0Then the branch is terminated to S9;
s8, at critical pointCSolving the number of curve branches based on the track curve branch rulegIf, ifgIf the number is equal to 1, the branch is not existed, the step is continuously skipped to S6 to continue the execution, and if the number is equal to 1, the step is continuously skipped togIf greater than 1, it indicates thatgBranch, then trace curve is dividedgEach branch is continuously expanded according to the rule of S6;
s9, if all branches of one line are terminated or merged into one branch, acquiring a track curve from a starting branch point to a final termination point or a merging point, and judging a legend corresponding to the track curve based on a track curve recognition modelD L The track curve recognition model is used for judging the linear shape of the track information between the two devices, and the step is S10;
s10, if all branches are stopped, jumping to S5;
s11, all the devices and characters in the graph are identified at the moment, then the center nodes and the side lengths of the devices and the characters are solved according to the track curve, and the corresponding relation between the characters and the devices is generated based on the position judgment rule.
Preferably, in the step S1, for the legendD L The specific process of acquiring the track information and the corresponding label of each device is as follows:
s11, drawing of a whole set of drawingsD L Firstly, finding all equipment images and corresponding equipment names, wherein the equipment names correspond to label information;
s12, aiming at each equipment image, starting from a starting pipeline connected with the equipment, randomly selecting a starting point, setting the starting point as an origin and coordinates as(0, 0)Obtaining the line width of the starting point informationdAnd withxAngle in positive direction of axisθThe information of each point ist={(x, y), d, θ}；
S13, starting from the starting point and along the line directionθExtending according to step 1, and if a line is forked, the line is also forked until the line cannot be extended and is just forkedThe branches are finally converged to reach the terminal point;
s14, the track from the starting point to the end point isT={t 1 , t 2 , …, t m }，mIndicating how many points there are, for each device,mis not fixed.
Preferably, the preprocessing operation is performed in the step S3 to obtain a drawing represented by a matrixM j The specific process of the pretreatment operation is as follows:
s31, for the drawingD j Firstly, obtaining the text content of the drawing, namely removing the frame and the right character part of the drawing, only keeping the instrument tube part, and using a matrix for the drawingM 1 Represents;
s32, for the matrixM 1 Traverse by rows, for each row, transverse linehIs made up ofh 1 , h 2 , h 1 Is composed of 3 segments arranged transversely, whereinh 1 Is longer than the thickest line segment,h 2 is 1.1 times of the distance between the dotted lines, and the transverse linehMove from left to right if both ends in each rowh1All partial values are greater than 0 and there is a part equal to 0, then this will beh 2 Changing the part of the matrix to be equal to 0 into 1, and obtaining the matrix after traversingM 2 ；
S33, for the matrixM 2 Traverse by column, for each column, vertical linevBy{v 1 , v 2 , v 1 }Is formed by vertically arranging 3 sections of lines, whereinv 1 Is longer than the thickest line segment,v 2 is 1.1 times of the distance between the dotted lines, and the transverse linevMove from top to bottom if both ends in each rowv 1 All partial values are greater than 0, and there is a portion equal to 0, then this will bevChanging 2 parts equal to 0 parts into 1 parts, and obtaining a matrix after traversing is finishedM 3 ；
S34, mixingMatrix ofM 3 Performing binarization processing to obtain a drawing represented by a matrixM j 。
Preferably, in the step S5, the line width is solved based on the boundary determination ruledDirection of the sum lineθThe boundary determination rule is specifically as follows:
s51, based on the pointsp(x 1 , y 1 ) Extending in both vertical and horizontal directions;
s52, when the matrix is encountered in the vertical direction or the horizontal directionM j If the z points are all equal to 0, the boundary part is indicated, 4 intersection points are obtained, and if the z points are all equal to 0, the extension continues;
s53, when the boundary part exists, aiming at each intersection point, the boundary part extends along two directions of the intersection pointmDot to obtainP i ，iThe number of the intersection points is shown,P i is composed ofmA set of points;
s54, for each setP i To is aligned withmAveraging the horizontal and vertical coordinates of the points to obtain four points{m 1 (x 1 , y 1 ), m 2 (x 2 , y 2 ), m 3 (x 3 , y 3 ), m 4 (x 4 , y 4 )}，m 1 Andm 2 is two intersection points in the vertical direction, and the vertical direction,m 3 andm 4 two intersection points in the horizontal direction;
s55, calculatingm 1 Andm 2 the distance betweend 1 ，m 3 Andm 4 the distance betweend 2 If it is determined thatd 1 Andd 2 a ratio of 10 or more, orLess than or equal to 0.1, the line is considered to be horizontal or vertical, and the line is takenmin( d 1 , d 2 )As the line width, andd 1 andd 2 the line where the longest line in the pair isxIncluded angle in the direction of positive half axisθAt this point, it is ended directly, ifd 1 Andd 2 if the ratio is less than 10 and greater than 0.1, the downward execution is continued;
s56, based on{m 1 (x 1 , y 1 ), m 2 (x 2 , y 2 ), m 3 (x 3 , y 3 ), m 4 (x 4 , y 4 )}Then two straight lines can be obtainedl u ：
Andl d ：
the slopes of the two straight lines are respectivelyk 1 Andk 2 in theory, inl u Andl d are two parallel lines, and in the figure, because pixel points cause errors, therefore, the pairk 1 Andk 2 averaging to obtain the slopekAngle with the horizontal directionθ =arctank；
S57, based on the slopekAdjustment ofl 1 Andl 2 to, form 1 Andm 3 averaging to obtainm u To is aligned withm 2 Andm 4 averaging to obtainm d Based on pointsm u And slopekTo obtainl' u Based on pointsm d And slope ofkTo obtainl' d Calculating by using a distance formula of two straight linesl' u Andl' d straight line distance therebetweend。
Preferably, the step S8 is performed at a critical pointCSolving the number of curve branches based on the track curve branch rulegThe track curve branching rule is specifically as follows:
s81, based on critical pointCThe direction of the point isθ c The width of the point isd c 4 intersections obtained based on the boundary decision rule are{mc 1 ((x 1 , y 1 ), mc 2 (x 2 , y 2 ), mc 3 (x 3 , y 3 ), mc 4 (x 4 , y 4 )}Critical point ofCAlong the edgeθ c The next point of direction isC'，The direction and width of the point are respectivelyθ' c Andd' c (ii) a 4 intersections obtained based on the boundary decision rule are{mc' 1 (x' 1 , y' 1 ), mc' 2 (x' 2 , y' 2 ), mc' 3 (x' 3 , y' 3 ), mc' 4 (x' 4 , y' 4 )}；
S82, ifd' c >> d c And among 4 intersections of Cmc' 1 ，mc' 2 The temperature of the molten metal is almost kept unchanged,mc' 3 ，mc' 4 a great change ofmc' 3 ，mc' 4 The temperature of the molten metal is almost kept unchanged,mc' 1 ，mc' 2 a great change of (2), the number of branchesgIs a number of the order of 2 or 3,the new expansion direction is
Or
While also possible expansion directions areθ c ；
S83, if 2-d c > d' c > d c ，θ c Andθ' c almost the same, and the coordinates of only one of the 4 intersections changes significantly, and the coordinates of the other 3 intersections hardly change, which indicates that there are diagonal branches, and the number of branches is 2, and it is assumed that the coordinates of the changed point areu'(x, y)Original coordinates of the intersection point areu(x', y')Based on pointsu'Extending along the boundary to obtain a line segmentl u' ，l u' For sets of line segments, using least-squares pairsl u' Fitting the points to obtain a straight linel 2 u' ，l 2 u' In the direction ofθ u' Based on points(cell((x+x')/2), cell((y+y')/2))Extension is carried out, if at allu'Above the point u, the direction of extension isθ u' If it is below, it isθ u' +180 °, the other branch direction is the original direction.
Preferably, in step S9, the trajectory information between the two devices is determined to be linear by using a trajectory curve recognition model, and a training process of the trajectory curve recognition model specifically includes:
s91, preparing track information of equipment and characters and corresponding labels, wherein the track information is from the equipment and the characters manually selected in legends and drawings, the labels correspond to the equipment information or the character information in the legends, and the labels of 'none' are added to all the labels to form a label libraryL，LIs of the size ofS L ；
S92, for each track informationT={t 1 , t 2 , …, t m }，t={(x, y), d, θ}}，Preprocessing is carried out, firstly, each point in T is matched withM' j IfM' j If the point is greater than 0, thentIs added with 1 as the last bit, i.e.t={(x, y), d, θ, 1}Otherwise, add 0, then pairTThe coordinate of each point in the data set is subtracted from the coordinate of the first point, and the preprocessed track information and the corresponding label form a data setDatasetAnd dividing the training set according to the proportion of 7TrainAnd test setTest；
S93, initializing a track curve recognition model, setting the learning rate to be 0.001, and setting an optimizer to beAdamWSetting a loss function as a cross entropy loss function;
s94, performing E times of iteration processes, wherein each iteration is from S95 to S97;
s95, randomly selecting from TrainBatch sizeThe data of the size is filled to a fixed length 512 by using 0, and is input into a track curve recognition model, and the output of the model is obtainedO；Batch sizeRepresenting the number of samples which are transmitted to the model for training in a single time;
s96, calculating the loss of the model based on the cross entropy loss functionAdamWOptimizing a model optimization model by an optimizer;
and S97, testing the effect of the model on the Test set every 10 iterations, and storing the model with the best Test effect.
Preferably, in step S11, the correspondence between the text and the device is determined based on a position determination rule, and the specific process of the position determination rule is as follows:
s111, for equipment and characters, based on the track information, solving coordinates of points at the top, bottom, left and right in the track information, and based on a rectangle formed by the four points, solving coordinates of a central point of the rectangle;
s112, for all the characters, calculating the distance between the central points, and if the distance is less than the threshold valueth 1 If the two characters belong to the same text, after calculating all the distances between the central points and determining whether the two characters can be combined or not, finally combining the two characters into the text and calculating the central point of the text;
s113, calculating the distance between the equipment and the text center point, and if the distance is less than a threshold valueth 2 And is the minimum distance from the text, the text is considered as the name of the device.
The invention provides a pipeline meter flow chart information extraction device based on deep learning, which comprises at least one processor and at least one memory, wherein the processor is coupled with the memory; the memory is used for storing a computer program of the pipeline meter flowchart information extraction method in the first aspect; the processor, when executing the computer program stored in the memory, causes the apparatus to perform a method for pipeline meter flow chart information extraction.
A third aspect of the present invention provides a computer-readable storage medium in which a program or instructions of the pipe meter flowchart information extraction method according to the first aspect are stored, which when executed by a processor, cause a computer to execute the pipe meter flowchart information extraction method.
Has the advantages that: compared with the prior art, the invention provides a pipeline instrument flow chart information extraction method and equipment based on deep learning, which reduces the time for CAD redrawing of PDF drawings and has the following beneficial effects:
1. in order to solve the problem that a lot of time is spent because manual redrawing is needed in the process of converting PDF drawings or hand-drawn drawings into CAD drawings, the invention provides a pipeline meter flow chart information extraction method based on deep learning, which can intelligently identify equipment and label information in the drawings and correspond the equipment and the label information in the pipeline meter flow chart.
2. When the information of the pipeline instrument flow chart is extracted, the invention innovatively provides a track generation and iteration method of equipment, pipelines, characters and the like of the pipeline instrument flow chart, and the method can iterate along the pipelines in the flow chart and generate track information of different equipment, pipelines and characters.
3. In order to solve the problems of poor linear detection effect and low identification accuracy for devices of different sizes when the whole drawing is subjected to target detection, the invention provides a track curve identification model, which uses a coding layer to code a track curve, uses a self-attention layer to extract the characteristics of the whole curve, and finally uses a classifier to obtain the device type.
Drawings
FIG. 1 is a flow chart of a PID map information extraction method constructed by the invention.
Fig. 2 is a schematic diagram of a binarized representation of an image.
FIG. 3 is a schematic diagram of a part of a legend and corresponding labeling information extracted from a PID diagram according to the present invention.
FIG. 4 is a schematic diagram of a part of a drawing and corresponding labeled information taken from a PID diagram according to the present invention.
FIG. 5 is a detailed illustration of the present invention with the dashed lines in the drawing turned into solid lines.
FIG. 6 is a diagram illustrating the boundary decision rule in detail according to the present invention.
FIG. 7 is a diagram illustrating the branching rules of the trajectory curves in detail according to the present invention.
FIG. 8 is a block diagram of a trajectory profile recognition model of the present invention.
Fig. 9 is a simple block diagram of a structure of the pipeline meter flowchart information extraction device based on deep learning according to the present invention.
Detailed Description
The invention is further illustrated by the following examples.
Example 1:
in this embodiment, a specific work flow of the method for extracting pipeline meter flowchart information based on deep learning according to the present invention is described with reference to fig. 1 to 8 based on a specific pipeline meter flowchart:
s1, for the legendD L Acquiring track information and a corresponding label of each device; PID drawing for a whole set of pipeline instrument flow chartD={{D 1 , …, D j ,…, D n }, D L }In whichD j Is a drawing in the drawing sheet D,nthe number of drawings is shown and indicated,D L is a legend, a legendD L Stipulate the drawingsDWhat each symbol represents, taking fig. 3 as an example, the specific process of acquiring the track information and the corresponding tag of each device is as follows:
s11, drawing of a whole set of drawingsD L Firstly, finding all the equipment images and the corresponding equipment names, wherein the equipment names correspond to the label information, and the circular frame in the figure 3 is the equipment and the rectangular frame is the equipment name;
s12, aiming at each equipment image, starting from a starting pipeline connected with the equipment, randomly selecting a starting point, setting the starting point as an origin point and coordinates as(0, 0)That is, the starting point in the circular frame in FIG. 3, the line width of the starting point information on the line is obtaineddAnd withxAngle in positive direction of axisθThe line width d and the included angle areθThe width and direction of the line where the red dot is located in the blue frame, the information of each dot ist={(x, y), d, θ}；
S13, starting from the starting point and along the line directionθ(i.e., horizontally in fig. 3), extend by step 1, and diverge (branch) when a line divergence (branch) is encountered (divergence point in the circular block diagram of fig. 3), until it cannot extend and all divergence eventually converge, reaching the endpoint (endpoint in the circular block diagram of fig. 3);
s14, the track from the starting point to the end point isT={t 1 , t 2 , …, t m }(black lines from start to end of the device within the circle in figure 3),mindicating how many points there are, for each device,mis not fixed.
S2, from{D 1 , …, D i ,…, D n }Any one of the drawingsD j Performing an iterative process from S3 to S11;
s3, for the drawingD j Taking FIG. 5 as an example, first a matrix representation of a drawing is obtainedM' j Performing preprocessing operation to obtain a drawing represented by a matrixM j FIG. 2 is an example of a drawing matrix representation; the specific process of the pretreatment operation is as follows:
s31, for the drawingD j Firstly, obtaining the text content of the drawing, namely removing the frame and the right text part of the drawing, only reserving the instrument tube part, and using a matrix for drawingM 1 Represents;
s32, the broken line in fig. 4 is processed, the horizontal broken line is changed to a solid line, and the matrix is described with reference to fig. 5 (a) and (b)M 1 The traversal is by rows, and for each row,his formed byh 1 , h 2 , h 1 And 3 sections of lines are transversely arranged, whereinh 1 Is longer than the thickest line segment,h 2 is 1.1 times of the distance between the dotted lines, and the transverse linehMove from left to right if two ends h in each row 1 All partial values are greater than 0, and there is a portion equal to 0, then this will beh 2 The portion partially equal to 0 is changed to 1 as shown in FIG. 5 (a), and in FIG. 5 (b), since the right end is locatedh 1 The part is 0, so the filling can not be carried out, and the matrix is obtained after the traversal is finishedM 2 ；
S33, now, the dotted line in fig. 4 is processed, the vertical dotted line is changed to a solid line, and the description is given with reference to fig. 5 (c) (d), where the matrix isM 2 A column-wise traversal is performed, for each column,vby{v 1 , v 2 , v 1 }3 sections of the wire are vertically arranged, whereinv 1 Is longer than the thickest line segment,v 2 is 1.1 times of the distance between the dotted lines, and the transverse linevMove from top to bottom if both ends in each rowv 1 All partial values are greater than 0 and there is a part equal to 0, then this will bev2 parts equal to 0Instead, 1, as in FIG. 5 (c), in FIG. 5 (d), due to the lower endv 2 The part is 0, so the filling can not be carried out, and the matrix is obtained after the traversal is finishedM 3 ；
S34, converting the matrixM 3 The binarization process is performed and the processing result is returned, one possible processing result being shown in fig. 2.
S4, to the matrixM j Performing an iterative process from S5 to S11;
s5, FIG. 6 (a) is a broken line cut from the instrument line diagram, FIG. 6 (b) is an edge shape of the line segment enlarged, and it can be seen that the edge appears jagged, and therefore, the edge appears in the matrixM j The edges of the median line are not smooth straight lines but rather irregular line segments. FromM j In which a point other than 0 is randomly selectedp(x 1 , y 1 ) Points of FIG. 4pThen, the line width is calculated from the vertical direction and the horizontal direction of the point based on the boundary determination ruledDirection of the sum lineθ(andxangle in the positive direction of the axis) ifM j If the sum is equal to 0, jumping to S11, wherein the specific process of the boundary decision rule is as follows:
s51, in FIG. 6 (c), based on the pointp(x 1 , y 1 ) Extending in both vertical and horizontal directions;
s52, when the matrix is encountered in the vertical direction or the horizontal directionM j When the point is equal to 0, the point extends along the same directionzPoint if thiszAll the points are equal to 0, if the points are in the boundary part, 4 intersection points are obtained, and if not, the extension is continued;
s53, when in the boundary part, for each intersection, extending along two directions of the boundary of the intersectionmDot, getP i ，iThe number of the intersection points is shown,P i is composed ofmSet of points, e.g. on the lower border in FIG. 6 (b)P 2 Shown, it consists of 10 points extending over the boundary;
s54, for each setP i To, formAveraging the horizontal and vertical coordinates of the points to obtain four points{m 1 (x 1 , y 1 ), m 2 (x 2 , y 2 ), m 3 (x 3 , y 3 ), m 4 (x 4 , y 4 )}，m 1 Andm 2 are two intersection points in the vertical direction,m 3 andm 4 the positional relationship of the four intersections is shown in fig. 6 (c), which is two intersections in the horizontal direction.
S55, calculatingm 1 Andm 2 the distance betweend 1 ，m 3 Andm 4 the distance betweend 2 If, ifd 1 Andd 2 the ratio of the above-mentioned components is not less than 10, or not more than 0.1, the line is considered horizontal or vertical, takemin( d 1 , d 2 )As the line width, andd 1 andd 2 the line on which the longest of the lines is located andxincluded angle in the direction of positive half axisθAt this point, it is finished directly, ifd 1 Andd 2 if the ratio is less than 10 and greater than 0.1, the downward execution is continued;
s56, based on{m 1 (x 1 , y 1 ), m 2 (x 2 , y 2 ), m 3 (x 3 , y 3 ), m 4 (x 4 , y 4 )}Then two straight lines can be obtainedl u ：
Andl d ：
the slopes of the two straight lines are respectivelyk 1 Andk 2 as indicated by the dotted line in FIG. 6 (c)l u Andl d as shown, since in theoryl u Andl d are two parallel lines, and in the figure, because pixel points cause errors, therefore, the pairk 1 Andk 2 averaging to obtain the slopekAngle with the horizontal directionθ =arctank；
S57, based on the slopekAdjustment ofl 1 Andl 2 to, form 1 Andm 3 averaging to obtainm u To, form 2 Andm 4 averaging to obtainm d As indicated by the dots in FIG. 6 (c)m u Andm d shown on a point basism u And slopekTo obtainl' u Based on pointsm d And slope ofkTo obtainl' d Calculating by using the distance formula of two straight linesl' u Andl' d straight-line distance therebetweend。
S6, based on the direction of the line of the previous pointθ，According to the formulax 2 = cell(x 1 + s*cosθ), y 2 = cell(y 1 + s*sinθ)Solving the coordinates of the next pointq(x 2 , y 2 )E.g. red dots in fig. 4q，sThe step size for each of the movements is indicated,cellindicating rounding up, setpointp 1 Top 2 Is branch B, judgeqWhether or not it has been walked by other branchesIf q isNot in the curve of the walk-through, thenEntering S7; if it is notqIn the curve that has already been walked, terminate the branch, jump to S9;
s7, finding out points with line width changed, such as the points in figure 4C. If it is notM j [x 2 , y 2 ]If the value is greater than 0, solving the point based on the boundary judgment ruleqLine width ofd’Direction of the sum lineθ’If, ifd’Is equal todSetting up a matrixM j MidpointpToqWidth ofdIs 0 while jumping to S6, otherwise the step size is reducedsRecalculatingqUntil founddIs not equal tod’Critical point ofC, entering intoS8; if it is notM j [x 2 , y 2 ]Less than or equal to 0, reducing step lengthsSkipping to S6; if when it is usedsWhen the value is equal to 1, the reaction solution is,M j [x 2 , y 2 ]and alsoIf the value is less than or equal to 0, the branch is terminated to enter S9;
s8, at the critical pointCHere, as the point in FIG. 4CSolving the number of curve branches based on the rule of the curve branches of the trackg(FIG. 4)CTwo branches at the point), ifgIf the number is equal to 1, the branch is not existed, the step is continuously skipped to S6 to continue the execution, and if the number is equal to 1, the step is continuously skipped togIf greater than 1, it indicates thatgEach branch, then the trace curve is divided outgEach branch is continuously expanded according to the rule of S6;
s81, based on critical pointCThe direction of the point isθ c The width of the point isd c 4 intersections obtained based on the boundary decision rule are{mc 1 ((x 1 , y 1 ), mc 2 (x 2 , y 2 ), mc 3 (x 3 , y 3 ), mc 4 (x 4 , y 4 )}Critical point ofCAlong the edgeθ c The next point of direction isC'，The direction and width of the point are respectivelyθ' c Andd' c (ii) a 4 intersections obtained based on the boundary decision rule are{mc' 1 (x' 1 , y' 1 ), mc' 2 (x' 2 , y' 2 ), mc' 3 (x' 3 , y' 3 ), mc' 4 (x' 4 , y' 4 )}As shown in fig. 7;
s82, as shown in FIG. 7 (b), when it is a vertical branch or a vertical branch, ifd' c >> d c And C' among the 4 intersectionsmc' 1 ，mc' 2 The temperature of the molten metal is almost kept unchanged,mc' 3 ，mc' 4 a great change ofmc' 3 ，mc' 4 The temperature of the molten metal is almost kept unchanged,mc' 1 ，mc' 2 is greatly changed, then the number of branchesg2 or 3, the new development direction is
Or
While also possible expansion directions areθ c ；
S83, as shown in FIG. 5 (a), when the branch is a diagonal branch, if 2 stard c > d' c > d c ，θ c Andθ' c almost the same, and the coordinates of only one of the 4 intersections changes significantly, and the coordinates of the other 3 intersections hardly change, which indicates that there are diagonal branches, and the number of branches is 2, and it is assumed that the coordinates of the changed point aremc' 2 (x', y')The original coordinates of the intersection point aremc 2 (x, y)Based on pointsmc' 2 Extending in both directions along the boundary4mDot to obtainP mc'2 ，P mc'2 Is on the boundary4mSet of points, as in FIG. 7 (a)P mc'2 Shown, using least squares methodP mc'2 Fitting the points to obtain a straight linel mc'2 ，l mc'2 In the direction ofθ mc'2 Based on points(cell((x+x')/2), cell((y+y')/2))Performing extension if pointu'Above the point u, the direction of extension isθ u' If it is below, it isθ u' +180 °, the other branch direction is the original direction.
S9, if all branches of one line are terminated or merged into one branch, acquiring a track curve from a starting branch point to a final termination point or a merging point (excluding the track curve which is detected), and judging a corresponding legend of the track curve based on a track curve recognition modelD L That of (1), e.g. the point in fig. 4aBegin bifurcation to convergence pointbForming a closed track curve loop; meanwhile, judging the linear shape of the track information between the two devices by using a track curve recognition model, and entering S10; as shown in the middle point of fig. 4qPoint toaAnd (4) point. The track curve identification model mainly comprises an encoding layer, a self-attention layer and a classification layer, and the specific result is shown in fig. 8. The track curve recognition method comprises the following steps that an encoding layer is used for encoding a track curve and converting the track curve into vector representation, information of the whole track is extracted from a user of an attention layer, a classification layer is mainly used for judging the equipment to which the track belongs, and the training process of a track curve recognition model is as follows:
s91, preparing track information of equipment and characters and corresponding labels, wherein the track information is from the equipment and the characters manually selected in legends and drawings, the labels correspond to the equipment information or the character information in the legends, and the labels of 'none' are added to all the labels to form a label libraryL，LIs of a size ofS L ；
S92, for each track informationT={t 1 , t 2 , …, t m }，t={(x, y), d, θ}}，Preprocessing, namely firstly, carrying out pretreatment on each point in TM' j If the real situation in (1) is comparedM' j If the point is greater than 0, thentIs added with 1 as the last bit, i.e.t={(x, y), d, θ, 1}Negation adding 0, then onTThe coordinates of the first point are subtracted from the coordinates of each point in the image, and the preprocessed track information and the corresponding label form a data setDatasetThe training set is divided according to the proportion of 7TrainAnd test setTest；
S93, initializing a track curve recognition model, setting the learning rate to be 0.001, and setting an optimizer to beAdamWSetting a loss function as a cross entropy loss function;
s94, performing E times of iteration processes, wherein each iteration is from S95 to S97;
s95, randomly selecting from TrainBatch sizeThe data of the size is filled to a fixed length 512 by using 0, and is input into a track curve identification model, and the output of the model is obtainedO；Batch sizeRepresenting the number of samples which are transmitted to the model for training in a single time;
s96, calculating the loss of the model based on the cross entropy loss functionAdamWOptimizing a model optimization model by an optimizer;
s97, testing the effect of the model on the Test set every 10 iterations, and storing the model with the best Test effect;
s10, if all branches are stopped, jumping to S5;
s11, all the devices and characters in the graph are identified at the moment, then the center nodes and the side lengths of the devices and the characters are solved according to the track curve, and the corresponding relation between the characters and the devices is judged based on the position judgment rule. The specific process of the position determination rule is as follows:
s111, solving coordinates of the uppermost point, the lowermost point, the leftmost point and the rightmost point in the track information based on the track information for the equipment and the characters, and solving coordinates of a central point of a rectangle based on the rectangle formed by the four points;
s112, for all characters, calculating the distance between the central points, if the distance is less than a threshold valueth 1 If the two characters belong to the same text, calculating the distances between all the central points and determining whether the two characters can be combined, finally combining the two characters into the text, and calculating the central point of the text;
s113, calculating the distance between the equipment and the center point of the text, and if the distance is smaller than a threshold valueth 2 And is the minimum distance from the text, the text is considered as the name of the device.
Example 2:
as shown in fig. 9, the present invention also provides a deep learning-based pipeline meter flowchart information extraction device, which includes at least one processor and at least one memory, as well as a communication interface and an internal bus; the memory stores computer executive programs; when the processor executes the execution program stored in the memory, the device is caused to execute the deep learning-based pipeline meter flowchart information extraction method in embodiment 1. The internal bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (enhanced Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the buses in the figures of the present application are not limited to only one bus or one type of bus. The memory may include a high-speed RAM memory, and may further include a non-volatile storage NVM, such as at least one magnetic disk memory, and may also be a usb disk, a removable hard disk, a read-only memory, a magnetic disk or an optical disk.
The device may be provided as a terminal, server, or other form of device.
Fig. 9 is a block diagram of an apparatus shown for illustration. The device may include one or more of the following components: processing components, memory, power components, multimedia components, audio components, interfaces for input/output (I/O), sensor components, and communication components. The processing components typically control overall operation of the electronic device, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components may include one or more processors to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component may include one or more modules that facilitate interaction between the processing component and other components. For example, the processing component may include a multimedia module to facilitate interaction between the multimedia component and the processing component.
The memory is configured to store various types of data to support operations at the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phonebook data, messages, pictures, videos, and so forth. The memory may be implemented by any type or combination of volatile and non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
The power supply component provides power to various components of the electronic device. The power components may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for an electronic device. The multimedia component includes a screen providing an output interface between the electronic device and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device is in an operating mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
The audio component is configured to output and/or input an audio signal. For example, the audio component includes a Microphone (MIC) configured to receive an external audio signal when the electronic device is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may further be stored in a memory or transmitted via a communication component. In some embodiments, the audio assembly further comprises a speaker for outputting audio signals. The I/O interface provides an interface between the processing component and a peripheral interface module, which may be a keyboard, click wheel, button, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.
The sensor assembly includes one or more sensors for providing various aspects of status assessment for the electronic device. For example, the sensor assembly may detect an open/closed state of the electronic device, the relative positioning of components, such as a display and keypad of the electronic device, the sensor assembly may also detect a change in position of the electronic device or a component of the electronic device, the presence or absence of user contact with the electronic device, orientation or acceleration/deceleration of the electronic device, and a change in temperature of the electronic device. The sensor assembly may include a proximity sensor configured to detect the presence of a nearby object in the absence of any physical contact. The sensor assembly may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly may further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device may access a wireless network based on a communication standard, such as WiFi,2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further comprises a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the electronic device may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic components for performing the above-described methods.
Example 3:
the present invention also provides a non-transitory computer-readable storage medium, in which a program or instructions are stored, and when the program or instructions are executed by a processor, the program or instructions cause a computer to execute a deep learning-based pipeline meter flowchart information extraction method according to embodiment 1.
In particular, a system, apparatus or device may be provided which is provided with a readable storage medium on which software program code implementing the functionality of any of the embodiments described above is stored and which causes a computer or processor of the system, apparatus or device to read out and execute instructions stored in the readable storage medium. In this case, the program code itself read from the readable medium can realize the functions of any of the above-described embodiments, and thus the machine-readable code and the readable storage medium storing the machine-readable code form part of the present invention.
The storage medium may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks (e.g., CD-ROM, CD-R, CD-RW, DVD-20 ROM, DVD-RAM, DVD-RW), magnetic tape, or the like. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.
It should be understood that the Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor, or in a combination of hardware and software modules.
It should be understood that a storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be integral to the processor. The processor and the storage medium may reside in an Application Specific Integrated Circuits (ASIC). Of course, the processor and the storage medium may reside as discrete components in a terminal or server.
The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.
The computer program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the disclosure are implemented by personalizing an electronic circuit, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), with state information of computer-readable program instructions, which can execute the computer-readable program instructions.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Although the present invention has been described with reference to the specific embodiments, it should be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention.
Claims (9)
1. A pipeline meter flow chart information extraction method based on deep learning is characterized by comprising the following steps:
s1, for the legendD L Acquiring track information and a corresponding label of each device; PID drawing for a whole set of pipeline instrument flow chartD={{D 1 , …, D j ,…, D n }, D L }WhereinD j Is a drawing in the drawing sheet D,nthe number of drawings is shown to indicate the number of drawings,D L is a legend, a legendD L Stipulate the drawingsDWhat each symbol in (1) represents;
s2, from{D 1 , …, D j ,…, D n }Any one of the drawingsD j Performing an iterative process from S3 to S11;
s3, for the drawingD j First, a matrix representation of the drawing is obtainedM' j Performing preprocessing operation to obtain a drawing represented by a matrixM j ；
S4, to the matrixM j Carrying out an iterative process from S5 to S10;
s5, fromM j Randomly selecting points other than 0p(x 1 , y 1 ) Then, the line width is extended from the vertical direction and the horizontal direction of the point, and the line width is solved based on the boundary determination ruledDirection of the sum lineθIf it is determined thatM j If the sum is equal to 0, jumping to S11;
s6, along the line of the last point in the drawing, according to the step lengthsMove to the next pointq(ii) a Direction of line based on last pointθ，According to the formulax 2 = cell(x 1 + s*cos θ), y 2 = cell(y 1 + s*sin θ)Solving the coordinates of the next pointq (x 2 , y 2 )，sThe step size for each of the movements is indicated,cellrepresents rounding up, setpointpToqIs branch B, judgeqWhether or not to alreadyIs walked by other branchesIf q isIf the curve is not in the walking curve, the step S7 is carried out; if it is notqIn the curve that has already been walked, terminate the branch, jump to S9;
s7, finding out points with changed line widths; if it is notM j [x 2 , y 2 ]If the value is greater than 0, solving the point based on the boundary judgment ruleqLine width ofd’Direction of the sum lineθ’If, ifd’Is equal todSetting up a matrixM j MidpointpToqWidth ofdIs 0 while jumping to S6, otherwise the step size is reducedsRecalculatingqUntil it is founddIs not equal tod’Critical point ofC, entering intoS8; if it is notM j [x 2 , y 2 ]If the value is less than or equal to 0, the step size is reducedsSkipping to S6; if it is whensWhen the value is equal to 1, the reaction solution is,M j [x 2 , y 2 ]and alsoIf the value is less than or equal to 0, the branch is terminated to enter S9;
s8, at critical pointCSolving the number of curve branches based on the track curve branch rulegIf, ifgIf the number is equal to 1, the branch is not existed, the step is continued to S6 for continuing execution, and if the branch is existed, the step is continuedgIf greater than 1, it indicates thatgBranch, then trace curve is dividedgEach branch is continuously expanded according to the rule of S6;
s9, if all branches of one line are terminated or merged into one branch, acquiring a track curve from a starting branch point to a final termination point or a merging point, and judging a legend corresponding to the track curve based on a track curve recognition modelD L The track curve recognition model is used for judging the linear shape of the track information between the two devices, and the step S10 is carried out;
s10, if all branches are stopped, jumping to S5;
s11, all the devices and characters in the graph are identified at the moment, then the center nodes and the side lengths of the devices and the characters are solved according to the track curve, and the corresponding relation between the characters and the devices is generated based on the position judgment rule.
2. The pipeline meter flowchart information extraction method based on deep learning as claimed in claim 1, wherein in step S1, for the legendD L The specific process of acquiring the track information and the corresponding label of each device is as follows:
s11, drawing of a whole set of drawingsD L Firstly, finding all equipment images and corresponding equipment names, wherein the equipment names correspond to label information;
s12, aiming at each equipment image, starting from a starting pipeline connected with the equipment, randomly selecting a starting point, setting the starting point as an origin point and coordinates as(0, 0)Obtaining the line width of the starting point informationdAnd withxAngle in positive direction of axisθThe information of each point ist={(x, y), d, θ}；
S13, starting from the starting point and along the line directionθExtending according to step 1, and branching the same when line branching occurs until the extension cannot be performed and all branches are finally converged to reach the terminal point;
s14, the track from the starting point to the end point isT={t 1 , t 2 , …, t m }，mIndicating how many points there are, for each device,mis not fixed.
3. The pipeline meter flowchart information extraction method based on deep learning of claim 1, wherein preprocessing operation is performed in step S3 to obtain a drawing represented by a matrixM j The specific process of the pretreatment operation is as follows:
s31, for the drawingD j Firstly, obtaining the text content of the drawing, namely removing the frame and the right character part of the drawing, only keeping the instrument tube part, and using a matrix for the drawingM 1 Represents;
s32, for the matrixM 1 Traverse by rows, for each row, transverse linehIs formed byh 1 , h 2 , h 1 Is composed of 3 segments arranged transversely, whereinh 1 Is longer than the thickest line segment,h 2 is 1.1 times of the distance between the dotted lines, and the transverse linehMove from left to right if both ends in each rowh1All partial values are greater than 0 and there is a part equal to 0, then this will beh 2 Changing the part of the matrix which is partially equal to 0 into 1, and obtaining the matrix after traversingM 2 ；
S33, for the matrixM 2 Traverse by column, for each column, vertical linevBy{v 1 , v 2 , v 1 }3 sections of lines are vertically arranged to form the component, whereinv 1 Is longer than the thickest line segment,v 2 is 1.1 times of the distance between the dotted lines, and the transverse linevMove from top to bottom if both ends in each rowv 1 All partial values are greater than 0, and there is a portion equal to 0, then this will bevChanging the part of 2 parts which is equal to 0 into 1, and obtaining the matrix after traversingM 3 ；
S34, combining the matrixM 3 Performing binarization processing to obtain a drawing represented by a matrixM j 。
4. The deep learning-based pipeline meter flowchart information extraction method as claimed in claim 1, wherein in step S5, the line width is solved based on the boundary determination ruledDirection of the sum lineθThe boundary determination rule is specifically as follows:
s51, based on the pointsp(x 1 , y 1 ) Extending in both vertical and horizontal directions;
s52, when the vertical direction or the horizontal direction meets the matrixM j When the number of the points is equal to 0, z points extend along the same directionIf the z points are all equal to 0, the description is in the boundary part, 4 intersection points are obtained, and otherwise, the extension is continued;
s53, when the intersection points are located at the boundary part, the intersection points are extended along two directions of the boundary of the intersection points for each intersection pointmDot to obtainP i ，iThe number of the intersection point is shown as,P i is composed ofmA set of points;
s54, for each setP i To, formAveraging the horizontal and vertical coordinates of the points to finally obtain four points{m 1 (x 1 , y 1 ), m 2 (x 2 , y 2 ), m 3 (x 3 , y 3 ), m 4 (x 4 , y 4 )}，m 1 Andm 2 is two intersection points in the vertical direction, and the vertical direction,m 3 andm 4 two intersection points in the horizontal direction;
s55, calculatingm 1 Andm 2 a distance therebetweend 1 ，m 3 Andm 4 the distance betweend 2 If, ifd 1 Andd 2 the ratio of the above-mentioned components is not less than 10, or not more than 0.1, the line is considered to be horizontal or vertical, takenmin( d 1 , d 2 )As the line width, andd 1 andd 2 the line on which the longest of the lines is located andxincluded angle in the direction of positive half axisθAt this point, it is ended directly, ifd 1 Andd 2 if the ratio is less than 10 and greater than 0.1, the downward execution is continued;
s56, based on{m 1 (x 1 , y 1 ), m 2 (x 2 , y 2 ), m 3 (x 3 , y 3 ), m 4 (x 4 , y 4 )}Then two straight lines can be obtainedl u ：
Andl d ：
the slopes of the two straight lines are respectivelyk 1 Andk 2 in theoryl u Andl d are two parallel lines, and in the figure, because pixel points cause errors, therefore, the pairk 1 Andk 2 averaging to obtain the slopekAngle with the horizontal directionθ =arctank；
S57, based on the slopekAdjustment ofl 1 Andl 2 to is aligned withm 1 Andm 3 averaging to obtainm u To, form 2 Andm 4 averaging to obtainm d Based on pointsm u And slope ofkTo obtainl' u Based on pointsm d And slopekTo obtainl' d Calculating by using a distance formula of two straight linesl' u Andl' d straight-line distance therebetweend。
5. The method for extracting pipeline meter flowchart information based on deep learning as claimed in claim 1, wherein the step S8 is performed at a critical pointCSolving the number of curve branches based on the track curve branch rulegThe track curve branching rule is specifically as follows:
s81, based on critical pointCThe square of the pointXiang Weiθ c The width of the point isd c 4 intersections obtained based on the boundary decision rule are{mc 1 ((x 1 , y 1 ), mc 2 (x 2 , y 2 ), mc 3 (x 3 , y 3 ), mc 4 (x 4 , y 4 )}Critical point ofCAlong the edgeθ c The next point of direction isC'，The direction and width of the point are respectivelyθ' c Andd' c (ii) a 4 intersections obtained based on the boundary decision rule are{mc' 1 (x' 1 , y' 1 ), mc' 2 (x' 2 , y' 2 ), mc' 3 (x' 3 , y' 3 ), mc' 4 (x' 4 , y' 4 )}；
S82, ifd' c >> d c And C' among the 4 intersectionsmc' 1 ，mc' 2 The temperature of the molten metal is almost kept unchanged,mc' 3 ，mc' 4 a great change ofmc' 3 ，mc' 4 The temperature of the molten metal is almost kept unchanged,mc' 1 ，mc' 2 a great change of (2), the number of branchesgIs 2 or 3, the new expansion direction is
Or
While also possible expansion directions areθ c ；
S83, if 2-d c > d' c > d c ，θ c Andθ' c almost the same, and the coordinates of only one of the 4 intersections changes significantly, and the coordinates of the other 3 intersections hardly change, which indicates that there are diagonal branches, and the number of branches is 2, and it is assumed that the coordinates of the changed point areu'(x, y)The original coordinates of the intersection point areu(x', y')Based on pointsu'Extending along the boundary to obtain a line segmentl u' ，l u' For sets of line segments, pairs using least squaresl u' Fitting the points to obtain a straight linel 2 u' ，l 2 u' In the direction ofθ u' Based on points(cell((x+x')/2), cell((y+y')/2))Performing extension if pointu'Above the point u, the direction of extension isθ u' If it is below, it isθ u' +180 °, the other branch direction is the original direction.
6. The method for extracting pipeline meter flowchart information based on deep learning of claim 1, wherein in step S9, a trajectory curve recognition model is used to determine a linear shape of the trajectory information between two devices, and a training process of the trajectory curve recognition model specifically includes:
s91, preparing track information of equipment and characters and corresponding labels, wherein the track information is from the equipment and the characters manually selected in legends and drawings, the labels correspond to the equipment information or the character information in the legends, and the labels of 'none' are added to all the labels to form a label libraryL，LIs of a size ofS L ；
S92, for each track informationT={t 1 , t 2 , …, t m }，t={(x, y), d, θ}}，Preprocessing, namely firstly, carrying out pretreatment on each point in TM' j If the real situation in (1) is comparedM' j If the point is greater than 0, thentIs added with 1 as the last bit, i.e.t ={(x, y), d, θ, 1}Negation adding 0, then onTThe coordinate of each point in the data set is subtracted from the coordinate of the first point, and the preprocessed track information and the corresponding label form a data setDatasetAnd dividing the training set according to the proportion of 7TrainAnd test setTest；
S93, initializing a track curve recognition model, setting the learning rate to be 0.001, and setting an optimizer to beAdamWSetting a loss function as a cross entropy loss function;
s94, performing E times of iteration processes, wherein each iteration is from S95 to S97;
s95, randomly selecting from TrainBatch sizeThe data of the size is filled to a fixed length 512 by using 0, and is input into a track curve identification model, and the output of the model is obtainedO；Batch sizeRepresenting the number of samples which are transmitted to the model for training in a single time;
s96, calculating the loss of the model based on the cross entropy loss functionAdamWOptimizing a model optimization model by an optimizer;
and S97, testing the effect of the model on the Test set every 10 iterations, and storing the model with the best Test effect.
7. The method for extracting information from a flow chart of a pipeline meter based on deep learning as claimed in claim 1, wherein in step S11, the correspondence between characters and devices is determined based on a position determination rule, and the specific process of the position determination rule is as follows:
s111, for equipment and characters, based on track information, coordinates of points at the top, the bottom, the left and the right in the track information are solved, and coordinates of a center point of a rectangle are solved based on the rectangle formed by the four points;
s112, for all the characters, calculating the distance between the central points, and if the distance is less than the threshold valueth 1 If the two characters belong to the same text, after calculating all the distances between the central points and determining whether the two characters can be combined or not, finally combining the two characters into the text, and calculatingA center point of the text;
s113, calculating the distance between the equipment and the center point of the text, and if the distance is smaller than a threshold valueth 2 And is the minimum distance from the text, the text is considered as the name of the device.
8. The utility model provides a pipeline instrument flow chart information extraction equipment based on deep learning which characterized in that: the apparatus comprises at least one processor and at least one memory, the processor and memory coupled; the memory stores a computer program of the pipeline meter flow chart information extraction method according to any one of claims 1 to 7; and when the processor executes the computer program stored in the memory, the equipment realizes the information extraction of the pipeline meter flow chart.
9. A computer-readable storage medium, in which a program or instructions of the pipe meter flowchart information extraction method according to any one of claims 1 to 7 are stored, which when executed by a processor, cause a computer to implement information extraction of a pipe meter flowchart.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211552869.8A CN115620014B (en) | 2022-12-06 | 2022-12-06 | Pipeline instrument flow chart information extraction method and equipment based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211552869.8A CN115620014B (en) | 2022-12-06 | 2022-12-06 | Pipeline instrument flow chart information extraction method and equipment based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115620014A CN115620014A (en) | 2023-01-17 |
| CN115620014B true CN115620014B (en) | 2023-04-07 |
Family
ID=84880886
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211552869.8A Active CN115620014B (en) | 2022-12-06 | 2022-12-06 | Pipeline instrument flow chart information extraction method and equipment based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115620014B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116110071B (en) * | 2023-04-07 | 2023-09-12 | 济南大学 | Image format pipeline and instrument diagram pipeline identification method based on deep learning |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3660743A1 (en) * | 2018-11-30 | 2020-06-03 | Tata Consultancy Services Limited | Systems and methods for automating information extraction from piping and instrumentation diagrams |
| KR20200065613A (en) * | 2018-11-30 | 2020-06-09 | 도프텍(주) | Method of automatically recognizing and classifying information of design in imaged PID drawings |
| CN113378671A (en) * | 2021-05-31 | 2021-09-10 | 邵艳杰 | Deep learning-based PID (proportion integration differentiation) drawing analysis method, computer system and medium |
| CN114694165A (en) * | 2022-06-01 | 2022-07-01 | 济南大学 | Intelligent PID drawing identification and redrawing method |
| CN114863464A (en) * | 2022-07-08 | 2022-08-05 | 济南大学 | Second-order identification method for PID drawing picture information |
-
2022
- 2022-12-06 CN CN202211552869.8A patent/CN115620014B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3660743A1 (en) * | 2018-11-30 | 2020-06-03 | Tata Consultancy Services Limited | Systems and methods for automating information extraction from piping and instrumentation diagrams |
| KR20200065613A (en) * | 2018-11-30 | 2020-06-09 | 도프텍(주) | Method of automatically recognizing and classifying information of design in imaged PID drawings |
| CN113378671A (en) * | 2021-05-31 | 2021-09-10 | 邵艳杰 | Deep learning-based PID (proportion integration differentiation) drawing analysis method, computer system and medium |
| CN114694165A (en) * | 2022-06-01 | 2022-07-01 | 济南大学 | Intelligent PID drawing identification and redrawing method |
| CN114863464A (en) * | 2022-07-08 | 2022-08-05 | 济南大学 | Second-order identification method for PID drawing picture information |
Non-Patent Citations (3)
| Title |
|---|
| Eun-Seop Yu et al.Features Recognition from Piping and Instrumentation Diagrams in Image Format Using a Deep Learning Network.《Energies》.2019,1-19页. * |
| Shubham Paliwal et al.Digitize-PID: Automatic Digitization of Piping and Instrumentation Diagrams.《arXiv:2109.03794v1》.2021,1-13页. * |
| 马明等.工程图纸图像识别技术在数字化交付中的应用.《石油化工建设 》.2021,第63-65页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115620014A (en) | 2023-01-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9171204B2 (en) | Method of perspective correction for devanagari text | |
| US10769487B2 (en) | Method and device for extracting information from pie chart | |
| US20240078646A1 (en) | Image processing method, image processing apparatus, and non-transitory storage medium | |
| CN111709339B (en) | Bill image recognition method, device, equipment and storage medium | |
| US8666199B2 (en) | Gesture-based selection text recognition | |
| JP4676225B2 (en) | Method and apparatus for capturing electronic forms from scanned documents | |
| US8515185B2 (en) | On-screen guideline-based selective text recognition | |
| US8831381B2 (en) | Detecting and correcting skew in regions of text in natural images | |
| KR20130056309A (en) | Text-based 3d augmented reality | |
| CN111798360A (en) | Watermark detection method, watermark detection device, electronic equipment and storage medium | |
| CN113343740B (en) | Table detection method, device, equipment and storage medium | |
| Shivakumara et al. | New gradient-spatial-structural features for video script identification | |
| CN115620014B (en) | Pipeline instrument flow chart information extraction method and equipment based on deep learning | |
| CN112199545A (en) | Keyword display method and device based on picture character positioning and storage medium | |
| CN111985469A (en) | Method and device for recognizing characters in image and electronic equipment | |
| Turk et al. | Computer vision for mobile augmented reality | |
| JP4247190B2 (en) | Two-dimensional code recognition device and program | |
| KR20140137254A (en) | Terminal, server, system and method for providing location information using character recognition | |
| CN115359008A (en) | Display interface testing method and device, storage medium and electronic equipment | |
| EP4207745A1 (en) | Method for embedding image in video, and method and apparatus for acquiring planar prediction model | |
| WO2021142765A1 (en) | Text line detection | |
| CN113177542A (en) | Method, device and equipment for identifying characters of seal and computer readable medium | |
| CN111860315A (en) | Method, device and equipment for detecting form line and storage medium | |
| CN112559342A (en) | Method, device and equipment for acquiring picture test image and storage medium | |
| CN111476090A (en) | Watermark identification method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |