CN107622247B - Express waybill positioning and extracting method - Google Patents
Express waybill positioning and extracting method Download PDFInfo
- Publication number
- CN107622247B CN107622247B CN201710880981.7A CN201710880981A CN107622247B CN 107622247 B CN107622247 B CN 107622247B CN 201710880981 A CN201710880981 A CN 201710880981A CN 107622247 B CN107622247 B CN 107622247B
- Authority
- CN
- China
- Prior art keywords
- graph
- express waybill
- node
- processed
- nodes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for positioning and extracting an express waybill, which comprises the following steps: the method comprises the following steps: selecting a key area of a printing pattern or a character existing in a reference express waybill as a reference position, and representing a graph; step two: representing the express waybill to be processed based on a candidate key area obtained by image segmentation; step three: calculating the similarity between the express waybills to be processed and the reference express waybills according to the attributes of the graphs, matching the graphs, determining the position mapping from the reference express waybills to the express waybills to be processed, and realizing the positioning of the user interested area on the express waybills to be processed; step four: and according to the positioning result of the related information, performing post-processing operations such as image rotation, segmentation, character block relocation and the like, and finishing the extraction of the information such as the name, the telephone number, the address and the like of the addressee interested by the user in the express waybill.
Description
Technical Field
The invention belongs to the technical field of postal service, and relates to a method for positioning and extracting express waybills by drawing and matching.
Background
The target information areas in the express waybill, such as logistics company icons, bar codes, two-dimensional codes, addressee names, sender names, telephone numbers, postcodes and the like, contain rich logistics information, and packages and letters can be sent to correct destinations only by identifying the information. The position of the relevant information in the express waybill is automatically positioned, so that the interference of the complex background information of the express waybill can be removed, and the information of the relevant area can be quickly positioned and identified, so that the parcels and the letters can be correctly sorted.
However, according to different demands of different logistics companies, the formulated express waybill images have different forms, and the positions and the typesetting of addresses, recipients, bar codes and the like have great difference. In addition, other complex table lines, patterns, irrelevant character areas and other complex backgrounds exist on the express waybill image, so that the difficulty in positioning and extracting the position of relevant information in the express waybill is increased.
Disclosure of Invention
The invention aims to provide a method for positioning and extracting express waybills through graph representation and matching so as to solve the problem of obtaining important information areas in the express waybills.
The specific technical scheme for realizing the purpose of the invention is as follows:
a map-showing and matching express waybill positioning and extracting method is characterized in that: the method comprises the following specific steps:
step 1: selecting an express waybill as a reference express waybill, selecting a mark, a bar code, a mobile phone number, a receiving address and a receiver name on the express waybill as a key area, and establishing full connection of the key area to obtain a reference express waybill map;
step 2: selecting candidate key areas of the to-be-processed express waybills by adopting an image segmentation algorithm, establishing a corresponding relation according to the texture similarity of the candidate key areas and a reference express waybills key area, and establishing a full-link map similar to the reference express waybills key area according to the corresponding relation, wherein different candidate key areas corresponding to the same key area are not connected, so that an to-be-processed express waybills map is obtained;
and step 3: measuring the similarity of the attributes of the reference express waybill map and the to-be-processed express waybill map, finding a congruent composition which is most similar to the reference express waybill map in the to-be-processed express waybill map, realizing the correspondence of nodes in the two maps, and taking the congruent composition as the optimal matching of the reference express waybill map;
and 4, step 4: and according to the best matched isograph and the actual positions of the corresponding nodes in the reference express waybill graph in the respective images, obtaining a parameter transformation matrix mapped from the reference express waybill to the corresponding pattern of the to-be-processed express waybill, and according to the parameter transformation matrix, completing the positioning and extraction of the user filling information area on the to-be-processed express waybill.
The step 1 specifically comprises:
step a 1: selecting at least 5-8 patterns and character blocks which are different in the same straight line and have distinguishable characteristics from the reference express waybill by adopting a manual labeling method, taking the patterns and the character blocks as key areas of the reference express waybill, and adding labels to the key areas; meanwhile, manually marking the position of a logistics information filling area on a reference express waybill for positioning and extracting related information;
step a 2: representing the obtained key areas as nodes of the graph, and representing the mutual position relation among the key areas as the edges of the graph; establishing a full-connection undirected graph q of a key area of a reference express waybill; wherein:
the full-connection undirected graph q ═ V, E, Lv，Leω, φ), where V is a node of the graph corresponding to a key region of a reference courier waybill; e is the edge of the graph and corresponds to the interconnection relationship among the key areas of the reference express waybill; l isv，LeThe labels representing each node V ∈ V and edge E ∈ E, respectively, are used to identify different nodes in the graph, [ omega ] represents the node attribute for each node V, [ phi ] represents the structural attribute for each node V in the graph q, and the node and structural attribute are defined as follows:
1) node attribute ω; the node V represents a local image of a small block, SIFT is adopted to describe the node attribute of the graph, and SIFT features have good rotation and scale invariance to the local feature description of the image and stronger robustness to illumination; the node attributes are represented as: omegai＝{Ti， Vi∈V}，TiDenoted as node viSIFT texture features of (1);
Ti＝{fi1，fi2，...，fij}，j＝1，2，...，M (1)
where f is a 128-dimensional SIFT feature vector, fijDenotes viM is a positive integer, and represents a node viA characteristic dimension of (d);
2) structural attribute phi; phi denotes a node viStructural attributes, including two sub-attributes: ray cluster attribute Π, angle attribute Θ and node ViIs expressed as phii＝{Πi，Θi，vi∈V}；
Ray cluster attribute piiThe attribute is represented by a node viFor fixed end points, other adjacent points vjResulting ray cluster network ray edge eijThe attribute is represented as follows:
Πi＝{ei1，ei2，...，eij}，i，j＝1，2，...，N，i≠j (2)
angle attribute thetai，θ(eij，eik) Node v in the representationiIs a vertex, eijAnd eikAngle formed by connecting edges, node ViHas an included angle attribute expressed as viSet of edge clip angles theta for the verticesiExpressed as follows:
Θi＝{θ(eij，eik)}i，j，k＝1，2，...，N，i≠j，i≠k，j≠k (3)
the step 2 specifically comprises:
step b 1: firstly, image segmentation based on a graph is adopted for an express waybill image to be processed, an MST tree set region with similar pixel points is established, then a selective search algorithm is adopted for combining adjacent MST tree set regions according to matching similarity of gray scale, texture and shape of the MST tree set regions, and image small blocks with different sizes and pattern information are obtained; selecting small image blocks with relatively concentrated patterns and characters as candidate key areas of the express waybill image to be processed; step b 2: sorting the candidate key areas of the express waybill to be processed and the reference express waybill key area according to the candidate key area similarity pair, and screening the first 3 image blocks with the highest similarity as candidate nodes corresponding to the reference express waybill map node;
step b 3: establishing a full connection graph G of the express waybills to be processed for the candidate nodes with the corresponding relation established,
namely, the express waybill map to be processed is obtained; and a plurality of candidate nodes corresponding to the same reference express waybill node are not connected.
The step 3 specifically includes:
step c 1: firstly, selecting nodes and connecting edges with different labels in the to-be-processed express waybill graph G as isomorphism of a reference express waybill graphGraph g, wherein the nodes of graph g and graph q are in one-to-one correspondence, is represented as
Step c 2: calculating similar distances according to the attribute difference between the same composition G in the graph G and the reference express waybill graph q to obtain the same composition G which is most similar to the graph q in the graph GmOr isomorphic diagram gsm(ii) a The calculation of the similar distance specifically includes:
1) nodal texture similarity distance: for node V in graph g and graph qgAnd V, adopting neighbor matching to the detected SIFT feature points, and removing mismatching points according to the distance ratio of the nearest neighbor matching points to the next nearest neighbor to obtain the matching feature point pairs of corresponding nodes in the two graphs
The similarity distance between graph nodes is:
wherein
ωiRepresenting the texture of the graph g and graph q nodes respectively,
logarithm of feature points representing the matching of corresponding nodes, dT(i)∈[0，1]，
Representation node
And viA union of features of (1);
2) node structure similarity distance: the node structure similarity distance comprises corresponding included angles in the graphs g and q and similarity of edges connected with nodes, and is respectively called included angle similarity distance and ray cluster edge similarity distance;
the included angles are similar; in the drawings
And viThe similar distance of the included angle is represented by the cosine distance of the vector:
wherein d isΘ(i)∈[0，1]；
Corresponding side similarity distance: in the two graphs, the corresponding nodes are marked
And viThe ratio of the lengths of all corresponding edges of a connection is X ═ X1，x2，…，XnTherein of
| represents edge eijI represents the current node label, j represents the adjacent point of i, n represents the number of nodes connected with the node i, and the degree of dispersion of X is calculated to obtain the similar distance of all corresponding edges connected by corresponding nodes in the two graphs:
wherein EX and DX are respectively the mean value and variance of variable X; in the drawings
And viThe similar distances of the corresponding sides are:
wherein d isΠ(i)∈[0，1](ii) a According to the above description, the diagram g and the diagram q of the reference courierThe similarity distance of the corresponding nodes is defined as:
d(q(i)，g(i))＝dT(i)+dΘ(i)+dП(i) (8)
wherein d (q), (i), g (i)) ∈ [0,3]When similar distance calculation of the reference express waybill graph q and the graph g is carried out, and when corresponding nodes in the graph g and the graph q are absent or wrong in selection, an isomorphic sub-graph g with a smaller similar distance to the graph q is consideredsmThe candidate nodes which are missing or selected wrongly are called outliers, the number of the outliers is represented by outlierNum, and the similar distance of the outliers is calculated in the following mode;
similar distance from cluster point:
wherein d isNum(i)∈[0，1]，NqRepresenting the number of nodes V in the reference courier manifest graph q, the similar distances of graph g and graph q are represented as:
wherein c isi∈ {0, 1}, 0 represents that the point is an outlier, 1 represents a node meeting the threshold requirement, and the smaller the value of d (q, G) is, the greater the similarity of the two graphs is, so in the to-be-processed express waybill graph G, the graph G with the minimum similarity distance to the reference express waybill graph q ismOr gsmAnd as a final matching result of the express waybill map G to be processed and the reference express waybill map q:
D(q，G)＝argmin(d(q，g)) (11)
at this time, the final matching result graph gmOr gsmIs recorded as graph gr。
The step 4 specifically includes:
step e 1: extract graph grAnd carrying out neighbor matching on SIFT feature points of corresponding nodes in the graph q to obtain a feature point pair set of the corresponding nodes, filtering the set by adopting a random sampling consistency algorithm, and removing mismatching noise point pairs to obtainTo a set of node feature pairs; fusing each group of node feature pair set to obtain a matching result graph grAnd the set of pairs of feature points of graph q;
step e 2: adopting a Zhang camera calibration algorithm to establish position mapping transformation of the feature point pairs under different viewing angles according to the feature point pair set of the graph obtained in the step e1, and solving parameters of internal parameters, external parameters and distortion coefficients in the camera calibration algorithm through the corresponding relation of the known feature point pairs to obtain a parameter transformation matrix of the position mapping of the feature point pairs; obtaining a mapping model according to a Zhang calibration camera model, wherein the mapping model comprises the following formula:
the homogeneous coordinate form of the M represents coordinates (u, v,1) under a plane coordinate system, namely coordinates of a reference express waybill map to be processed, and the homogeneous coordinate form of the M represents coordinates (x, y,1) under a world coordinate system, namely coordinates of characteristic points on the reference express waybill map; [ R t]R ═ R as an external parameter of the camera1*r2*r2An image rotation matrix along the direction of XYZ coordinate axes is represented, and t represents a translation matrix; s denotes the scale factor, A denotes the camera intrinsic parameters, A [ R t ]]Namely, the mapping process from the reference express waybill diagram to be processed is shown; for the calibrated two-dimensional plane graph, the world coordinate system Z is 0, so the above formula can be converted into the following form:
wherein H is A [ r ]1r2t]And modifying A, r in H by fitting the characteristic point pair set to H by adopting a maximum likelihood estimation method1、r2T, improved
Namely a parameter transformation matrix of the mapping of the position of the characteristic point pair, namely a corresponding graph from the pixel position of the pattern on the reference express waybill to the reference express waybill diagram to be processedA parameter transformation matrix for mapping the pixel position;
step e 3: the position of a logistics information filling area manually marked on a reference express waybill is transformed by a parameter transformation matrix
Mapping to the express waybill to be processed to obtain the position of a logistics information filling area corresponding to the express waybill to be processed, and positioning the logistics information filling area filled by a user on the express waybill to be processed;
step e 4: and according to the positioned position and direction of the logistics information filling area filled by the user, performing rotary interpolation calculation on the express waybill to be processed to obtain a horizontal and upright text image, and completing the extraction of the logistics information filled by the user.
According to the method, the pattern areas with distinguishable characteristics, such as the logo, the bar code, the addressee, the address of the addressee, the zip code, the telephone number and the like of the logistics company, are extracted from the scene express waybill and the express waybill image according to the content defined on the reference express waybill template. Irrelevant image information is removed through a graph representation and graph matching method for the key area, so that interference is reduced, and the positioning accuracy is improved. The method defines the position of a key area serving as a reference datum position on a reference express waybill in a man-machine interaction mode, so that the obtained target image area has diversity, and compared with a supervised learning method, the method has stronger ductility and stronger universality. According to the method, SIFT feature matching is adopted to establish a node corresponding relation, and similarity measurement of a key region is carried out in a graph matching mode, so that the method has strong robustness on illumination, rotation, inclination, handstand and partially shielded images.
Drawings
FIG. 1 is a flow chart of the present invention;
FIG. 2 is an illustration of selecting key area samples with reference to an express waybill according to the present invention;
FIG. 3 is a diagram of a reference express waybill key area of the present invention;
FIG. 4 is a flow chart of a candidate key region segmentation process of an express waybill to be processed according to the present invention;
FIG. 5 is a schematic block diagram of the same composition construction process of the present invention;
FIG. 6 is a candidate key area diagram corresponding to a final matching result diagram of the to-be-processed thermal express waybill according to the present invention;
fig. 7 is a candidate key area diagram corresponding to a final matching result diagram of a multi-express waybill to be processed according to the present invention;
FIG. 8 is a schematic diagram of the positioning and extraction results of the information filling area of the to-be-processed thermal express waybill according to the present invention;
fig. 9 is a schematic diagram of positioning and extracting results of a to-be-processed multi-connected express waybill information filling area.
Detailed Description
The following detailed description of embodiments of the invention refers to the accompanying drawings.
The invention adopts a method based on graph representation and matching to position and extract express waybill, and figure 1 shows a flow chart of the method based on graph representation and matching, which is divided into four steps: selecting key areas such as printing patterns or characters and the like in a reference express waybill as reference positions, and representing the graph, wherein the key areas are selected and represented; 2) performing graph representation on the express waybill to be processed based on the candidate key area obtained by image segmentation, wherein the graph representation comprises key area selection, key area screening and graph representation; 3) calculating the similarity between the express waybills to be processed and the reference express waybills according to the attributes of the graphs, matching the graphs, determining the position mapping from the reference express waybills to the express waybills to be processed, and realizing the positioning of the user interested area on the express waybills to be processed; 4) and according to the positioning result of the related information, performing post-processing operations such as image rotation, segmentation, character block relocation and the like, and finishing the extraction of the information such as the name, the telephone number, the address and the like of the addressee interested by the user in the express waybill.
And selecting a key area of the express waybill according to the reference. Fig. 2 shows an example of selecting a key area from a postal parcel receipt, and when a reference express waybill diagram representation is established, a user manually selects an image area block capable of reflecting the express waybill characteristics, such as an express waybill company mark with distinguishable characteristics, a specific pattern, a character block and the like, and establishes a key area represented by full connection of the diagram. Considering that at least three points are needed to determine a plane, the user needs to select at least 3 image blocks with central points of the areas not on the same straight line when selecting the key area. Because the express waybill image has more characters and a complex background, enough key areas are needed to realize registration in the subsequent image matching process, and meanwhile, the matching calculation amount is moderate, 5-8 image blocks with complete, clear and moderate patterns are recommended to be selected as the key areas.
And (3) establishing a full-connection undirected graph representation of the reference express waybill shown in the figure 3(a) by taking the selected candidate key areas as graph nodes. Defining the undirected graph as q ═ V, E, Lv，Leω, φ), where V is a node of the graph corresponding to a key region of a reference courier waybill; e is the edge of the graph and corresponds to the interconnection relationship among the key areas of the reference express waybill; l isv，LeThe labels representing each node V ∈ V and edge E ∈ E, respectively, are used to identify different nodes in the graph, ω represents the node attribute for each node V, and φ represents the structural attribute for each node V in the graph q, FIG. 3(b) is a diagram of node V in FIG. 3(a)7Is represented by the structural attributes of (1). Wherein the nodes and the structural attributes are defined as follows:
1) node attribute ω; the node V represents a local image of a small block, SIFT is adopted to describe the node attribute of the graph, and SIFT features have good rotation and scale invariance to the local feature description of the image and stronger robustness to illumination; the node attributes are represented as: omegai＝{Ti， vi∈V}，TiDenoted as node viSIFT texture features of (1);
Ti＝{fi1，fi2，...，fij}，j＝1，2，...，M (1)
where f is a 128-dimensional SIFT feature vector, fijDenotes viM is a positive integer, and represents a node viA characteristic dimension of (d);
2) structural PropertiesPhi; phi denotes a node viStructural attributes, including two sub-attributes: ray cluster attribute Π, angle attribute Θ and node ViIs expressed as phii＝{Πi，Θi，vi∈V}；
Ray cluster attribute piiThe attribute is represented by a node viFor fixed end points, other adjacent points vjResulting ray cluster network ray edge eijThe attribute is represented as follows:
Πi＝{ei1，ei2，...，eij}，i，j＝1，2，...，N，i≠j (2)
angle attribute thetai，θ(eij，eik) Node v in the representationiIs a vertex, eijAnd eikAngle formed by connecting edges, node ViHas an included angle attribute expressed as viSet of edge clip angles theta for the verticesiExpressed as follows:
Θi＝{θ(eij，eik)}i，j，k＝1，2，...，N，i≠j，i≠k，j≠k (3)
the invention adopts a method for positioning and extracting express waybill based on graph representation and matching, firstly adopts graph-based image segmentation to express waybill images to be processed, and establishes a Minimum Spanning Tree (MST) of similar pixel points, wherein the MST establishing process comprises the following steps:
firstly, each pixel point of an image is regarded as a single area, each pixel point of the image is taken as an image vertex, the similarity between adjacent pixels is taken as a weight edge, the image pixel points are constructed into a weighted undirected graph, a proper pixel similarity threshold value is used for establishing a minimum spanning tree MST of the pixel points, and the obtained MST is the image area obtained by segmentation. The pixel similarity is the RGB Euclidean distance of the image, and the similarity threshold value adopts an adaptive threshold value, namely the intra-MST class distance is minimized, and the inter-MST class distance is maximized. When the scene image is divided by adopting an image based on a graph, calculating RGB Euclidean distances of two points of color and gray scale information, and expressing to measure the difference degree of the two points, wherein the RGB Euclidean distances are expressed as follows:
wherein pr, pg and pb are color image 3-channel pixel values, and the gray image only calculates Euclidean distance of single-channel pixel values. Wherein a threshold is set, and when the Euclidean distance (14) between two pixels is smaller than the threshold, the pixels are combined. Iterative merging, according to the basic idea of region growing, the image is finally merged into image segmentation regions, namely MST. Because the difference of the internal gray level of the image is large, the global threshold is not suitable for area segmentation, and the threshold can be selected in a self-adaptive mode according to the image. For example, the region denoted by MST is denoted by C, and the regions denoted by C are selectedi，CjI, j ═ 1, 2.. intra-class difference Int (C) according to regioni) And inter-class Diff (C) of regionsi，Ci) The adaptive threshold is adapted.
Intra-class difference Int (C)i)：
w(vi1，vi2) Two pixels v in a minimum spanning tree MST denoted by the reference iit，vikRGB Euclidean distance of (Int) (C)i) Definition of w (v)it，vik) The maximum value of (2) represents the pixel point with the maximum similar difference in the MST, i.e., the difference value of the maximum brightness or color in one region.
Inter-class Diff (C)i，Cj)：
I.e. in two areas Ci，GjAnd taking the pixel point with the minimum pixel difference degree between the regions as the similarity of the two regions.
Judging the standard of C combination:
Diff(Ci，Cj)≤min(Int(Ci)，Int(Cj)) (17)
wherein Int (C)i)，Int(Cj) Are respectively the region CiAnd CjMaximum acceptable difference, Diff (C) between current areasi，Cj) Than two areas Ci，CjIf the degree of difference between the two parts is smaller, merge Ci，CjThat is, two MSTs are merged to obtain the MST tree set region tr.
Next, the selective search algorithm shown in fig. 4(a) is used to merge the adjacent MST tree set regions according to their matching similarity of gray scale, texture, and shape, so as to obtain image patches with different sizes and pattern information. The selective search algorithm selects an MST tree set area with more abundant information as a primitive on the basis of image segmentation, and adopts a hierarchical merging strategy to construct a Huffman tree of the MST tree set area so as to obtain small image blocks merged at different levels.
The selective search algorithm:
1) using a graph-based image segmentation algorithm to obtain MST tree set Tr ═ { Tr ═ Tr1，tr2，...，trm}
2) Initializing a similarity set
3) Calculate two-by-two adjacent triAnd trjAdding the similarity into a similarity set S;
4) find out the two regions tr with the maximum similarity from the similarity set SiAnd trjMerge them into one region trtUpdating tr in R at the same timeiAnd trjIs trtWhile recalculating trtAnd (4) updating the set S according to the similarity with other areas.
5) Acquiring tr in the set SiThe image patches with different sizes and pattern information are obtained.
And selecting small image blocks with relatively concentrated patterns and characters as candidate key areas of the express waybill image to be processed. The selection strategy is as follows:
1) all the image small blocks obtained in the selective search algorithm process adopt Sobel operator to strengthen the edge to obtain the strengthened average gray GradientMEAN；
2) In GradientMEANAnd filtering out the segment with smaller average value of the gray gradient of the image patch as a gradient threshold value.
3) Image patch size filtering under the following screening conditions:
IMGw*0.4≥Sw≥32，，IMGh*0.15≥Sh≥32 (18)
wherein IMGwAnd IMGhIs the width and height, S, of the image to be processedwAnd ShIs the width and height of the image patch.
According to the selection process, candidate key areas of the to-be-processed express waybills are obtained, wherein the candidate key areas are shown in fig. 4 (b).
For express waybill images with large size and complex background, in the graph matching process, if all candidate regions are used as traversal nodes, the calculated amount is large. In order to improve the efficiency of matching the reference express waybill map, the first 3 image blocks with the highest similarity are screened out as candidate nodes corresponding to the nodes of the reference express waybill map according to the candidate key region sequence of the texture feature similarity pair of the candidate key region of the express waybill to be processed and the key region of the reference express waybill, a large number of candidate key regions with too low similarity are removed, the matching complexity is reduced, and the operation efficiency of the algorithm is improved. Establishing candidate nodes, establishing a full connection graph G of the to-be-processed express waybills shown in figure 5(b),
namely, the express waybill map to be processed is obtained; and a plurality of candidate nodes corresponding to the same reference express waybill node are not connected.
The isomorphic diagram is first defined. Suppose G1＝(V1，E1) And G2＝(V2，E2) Is two graphs, if there is a bijection σ: v1→V2So that for all x, y ∈ V1All have xy ∈ E1Equivalent to σ (x) σ (y) ∈ E2, such a mapping σ is referred to as an isomorphism, graph G1And G2The patterns are the same as each other. Selecting nodes and connecting edges with different labels in the to-be-processed express waybill graph G as the same graph G of the reference express waybill graph, wherein the nodes of the graph G and the graph q are in one-to-one correspondence, and representing the graph G as
According to the correspondence between the graph q and the key area in the graph G, in the to-be-processed express waybill graph G shown in fig. 5(b), according to the definition σ of the isomorphic mapping, the reference express waybill graph q shown in fig. 5 (a): the candidate corresponding to { a, b, c, d } is denoted as g in 5(c)1：{a1，b1，c1，d1}，...，g64：{a3，b3，c3，d3}。
Calculating similar distances according to the attribute difference between the same composition G in the graph G and the reference express waybill graph q to obtain the same composition G which is most similar to the graph q in the graph GmOr isomorphic diagram gsm(ii) a The calculation of the similar distance specifically includes:
1) nodal texture similarity distance: for node V in graph g and graph qgAnd V, adopting neighbor matching to the detected SIFT feature points, and removing mismatching points according to the distance ratio of the nearest neighbor matching points to the next nearest neighbor to obtain the matching feature point pairs of corresponding nodes in the two graphs
The similarity distance between graph nodes is:
wherein
ωiRepresenting the texture of the graph g and graph q nodes respectively,
logarithm of feature points representing the matching of corresponding nodes, dT(i)∈[0，1]，
Representation node
And viA union of features of (1);
2) node structure similarity distance: the node structure similarity distance comprises corresponding included angles in the graphs g and q and similarity of edges connected with nodes, and is respectively called included angle similarity distance and ray cluster edge similarity distance;
the included angles are similar; in the drawings
And viThe similar distance of the included angle is represented by the cosine distance of the vector:
wherein d isΘ(i)∈[0，1]；
Corresponding side similarity distance: in the two graphs, the corresponding nodes are marked
And viThe ratio of the lengths of all corresponding edges of a connection is X ═ X1，x2，…，XnTherein of
| represents edge eijI represents the current node label, j represents the adjacent point of i, n represents the number of nodes connected with the node i, and the degree of dispersion of X is calculated to obtain the similar distance of all corresponding edges connected by corresponding nodes in the two graphs:
wherein EX and DX are respectively the mean value and variance of variable X; in the drawings
And viThe similar distances of the corresponding sides are:
wherein d isn(i)∈[0，1](ii) a According to the above description, the similar distances of the corresponding nodes of the graph g and the reference express waybill graph q are defined as:
d(q(i)，g(i))＝dT(i)+dΘ(i)+dП(i) (8)
wherein d (q), (i), g (i)) ∈ [0,3]When similar distance calculation of the reference express waybill graph q and the graph g is carried out, and when corresponding nodes in the graph g and the graph q are absent or wrong in selection, an isomorphic sub-graph g with a smaller similar distance to the graph q is consideredsmThe candidate nodes which are missing or selected wrongly are called outliers, the number of the outliers is represented by outlierNum, and the similar distance of the outliers is calculated in the following mode;
similar distance from cluster point:
wherein d isNum(i)∈[0，1]，NqRepresenting the number of nodes V in the reference courier manifest graph q, the similar distances of graph g and graph q are represented as:
wherein c isi∈ {0, 1}, 0 indicates that the point is an outlier, 1 indicates a node meeting the threshold requirement, and a smaller value of d (q, G) indicates a greater similarity between the two graphs, so in the to-be-processed express waybill graph G, the to-be-processed express waybill graph G will be similar to the reference express waybill graphThe similar distance of the menu diagram q is the smallest diagram gmOr gsmAnd as a final matching result of the express waybill map G to be processed and the reference express waybill map q:
D(q，G)＝argmin(d(q，g)) (11)
at this time, the final matching result graph gmOr gsmIs recorded as graph gr。
Obtaining a isomorphic graph g which is best matched with the reference express waybill graph through graph similarity distance calculationmOr isomorphic diagram gsmFig. 6 and 7 respectively show the best matching results of the key areas of the thermosensitive express waybills to be processed and the multi-couple express waybills to be processed, wherein the same reference numerals on the express waybills to be processed and the express waybills to be processed indicate the matched corresponding key areas.
Extract graph grCarrying out neighbor matching on SIFT feature points of corresponding nodes in the graph q to obtain a feature point pair set of the corresponding nodes, filtering the set by adopting a random sampling consistency algorithm, and removing mismatching noise point pairs to obtain a node feature pair set; fusing each group of node feature pair set to obtain a matching result graph grAnd the set of pairs of feature points of graph q;
adopting a Zhang camera calibration algorithm according to the characteristic point pair set of the graph obtained in the process, establishing position mapping transformation of the characteristic point pairs under different visual angles, and solving internal parameter, external parameter and distortion coefficient parameters in the camera calibration algorithm through the corresponding relation of the known characteristic point pairs to obtain a parameter transformation matrix of the position mapping of the characteristic point pairs; obtaining a mapping model according to a Zhang calibration camera model, wherein the mapping model comprises the following formula:
the homogeneous coordinate form of the M represents coordinates (u, v,1) under a plane coordinate system, namely coordinates of a reference express waybill map to be processed, and the homogeneous coordinate form of the M represents coordinates (x, y,1) under a world coordinate system, namely coordinates of characteristic points on the reference express waybill map; [ R t]R ═ R as an external parameter of the camera1*r2*r2An image rotation matrix along the direction of XYZ coordinate axes is represented, and t represents a translation matrix; s denotes the scale factor, A denotes the camera intrinsic parameters, A [ R t ]]Namely, the mapping process from the reference express waybill diagram to be processed is shown; for the calibrated two-dimensional plane graph, the world coordinate system Z is 0, so the above formula can be converted into the following form:
wherein H is A [ r ]1r2t]And modifying A, r in H by fitting the characteristic point pair set to H by adopting a maximum likelihood estimation method1、r2T, is improved
The mapping matrix is a parameter transformation matrix of the mapping of the position of the characteristic point pair, namely a parameter transformation matrix of the mapping from the pattern pixel position on the reference express waybill to the pattern pixel position corresponding to the reference express waybill to be processed;
fig. 8 and 9 show the key region information locating and extracting processes of the to-be-processed heat-sensitive express waybill and the to-be-processed multi-gang express waybill respectively, and since the position for extracting the relevant information is defined on the reference express waybill, the location of the extracted key region information mapped on the to-be-processed express waybill image is located according to the obtained function mapping model (1), as shown by white locating boxes in fig. 8(a) and 9 (a). Because the direction of the acquired to-be-processed express waybill image is uncertain, according to parameters, the position of a key information area after mapping is possibly not a horizontally upright text line, the position and the direction information of the extracted key information can be manually marked on a template image, and rotary interpolation calculation is carried out to obtain a final horizontally upright text image, fig. 8(b) and fig. 9(b) respectively show the to-be-processed heat-sensitive express waybill and the image after the direction correction of the to-be-processed multi-express waybill, two groups of express waybill images shown in fig. 8 and fig. 9 have different resolutions, brightness, direction deflection, surface single fold and deformation, and the positioning and direction correction conditions of related areas in the groups (a) and (b) show that the text algorithm can well position the images with different image quality differences and different types. Because accurate positioning is guaranteed, characters of the express waybill area obtained by segmentation are complete, clear and accurate. In addition, simple character connected domain merging is performed on the image blocks obtained by dividing the express waybill, and the extracted information related to the express waybill is the address of the addressee, the name of the addressee and the telephone number respectively as shown in (c), (d) and (e) in fig. 8 and 9.
Claims (3)
1. A method for positioning and extracting an express waybill is characterized by comprising the following specific steps:
step 1: selecting an express waybill as a reference express waybill, selecting a mark, a bar code, a mobile phone number, a receiving address and a receiver name on the express waybill as a key area, and establishing full connection of the key area to obtain a reference express waybill map;
step 2: selecting candidate key areas of the to-be-processed express waybills by adopting an image segmentation algorithm, establishing a corresponding relation according to the texture similarity of the candidate key areas and a reference express waybills key area, and establishing a full-link map similar to the reference express waybills key area according to the corresponding relation, wherein different candidate key areas corresponding to the same key area are not connected, so that an to-be-processed express waybills map is obtained;
and step 3: measuring the similarity of the attributes of the reference express waybill map and the to-be-processed express waybill map, finding a congruent composition which is most similar to the reference express waybill map in the to-be-processed express waybill map, realizing the correspondence of nodes in the two maps, and taking the congruent composition as the optimal matching of the reference express waybill map;
and 4, step 4: according to the best matched isomorphic graph and the actual positions of the corresponding nodes in the reference express waybill graph in respective images, obtaining a parameter transformation matrix mapped from the reference express waybill to the corresponding pattern of the to-be-processed express waybill, and completing the positioning and extraction of the information filling area of the user on the to-be-processed express waybill according to the parameter transformation matrix; wherein:
the step 1 specifically comprises:
step a 1: selecting at least 5-8 patterns and character blocks which are different in the same straight line and have distinguishable characteristics from the reference express waybill by adopting a manual labeling method, taking the patterns and the character blocks as key areas of the reference express waybill, and adding labels to the key areas; meanwhile, manually marking the position of a logistics information filling area on a reference express waybill for positioning and extracting related information;
step a 2: representing the obtained key areas as nodes of the graph, and representing the mutual position relation among the key areas as the edges of the graph; establishing a full-connection undirected graph q of a key area of a reference express waybill; wherein:
the full-connection undirected graph q ═ V, E, Lv，Leω, φ), where V is a node of the graph corresponding to a key region of a reference courier waybill; e is the edge of the graph and corresponds to the interconnection relationship among the key areas of the reference express waybill; l isv，LeThe labels representing each node V ∈ V and edge E ∈ E, respectively, are used to identify different nodes in the graph, [ omega ] represents the node attribute for each node V, [ phi ] represents the structural attribute for each node V in the graph q, and the node and structural attribute are defined as follows:
1) node attribute ω; since the node V represents a local image of a small block, SIFT is used to describe the node attribute of the graph, and the node attribute is represented as: omegai＝{Ti，vi∈V}，TiDenoted as node viSIFT texture features of (1);
Ti＝{fi1,fi2,...,fij},j＝1,2,...,M (1)
where f is a 128-dimensional SIFT feature vector, fijDenotes viM is a positive integer, and represents a node viA characteristic dimension of (d);
2) structural attribute phi; phi denotes a node viStructural attributes, including two sub-attributes: ray cluster attribute Π and angle attribute Θ, node viIs expressed as phii＝{Πi,Θi,vi∈V}；
Ray cluster attribute piiThe attribute is represented by a node viFor fixed end points, other adjacent points vjResulting ray cluster network ray edge eijThe attribute is represented as follows:
Πi＝{ei1,ei2,...,eij},i,j＝1,2,...,N,i≠j (2)
angle attribute thetai，θ(eij，eik) Node v in the representationiIs a vertex, eijAnd eikAngle formed by connecting edges, node viHas an included angle attribute expressed as viSet of edge clip angles theta for the verticesiExpressed as follows:
Θi＝{θ(eij,eik)}i,j,k＝1,2,...,N,i≠j,i≠k,j≠k (3)；
the step 2 specifically comprises:
step b 1: firstly, image segmentation based on a graph is adopted for an express waybill image to be processed, an MST tree set region with similar pixel points is established, then a selective search algorithm is adopted for combining adjacent MST tree set regions according to matching similarity of gray scale, texture and shape of the MST tree set regions, and image small blocks with different sizes and pattern information are obtained; selecting small image blocks with relatively concentrated patterns and characters as candidate key areas of the express waybill image to be processed;
step b 2: sorting the candidate key areas of the express waybill to be processed and the reference express waybill key area according to the candidate key area similarity pair, and screening the first 3 image blocks with the highest similarity as candidate nodes corresponding to the reference express waybill map node;
step b 3: establishing a full connection graph G of the express waybills to be processed for the candidate nodes with the corresponding relation established,
namely, the express waybill map to be processed is obtained; and a plurality of candidate nodes corresponding to the same reference express waybill node are not connected.
2. The express waybill positioning and extracting method of claim 1, wherein the step 3 specifically comprises:
step c 1: firstly, selecting nodes and connecting edges with different labels in an express waybill graph G to be processed as an identical graph G of a reference express waybill graph, wherein the nodes of the graph G and the graph q are in one-to-one correspondence, and representing the graph G as
Step c 2: calculating similar distances according to the attribute difference between the same composition G in the graph G and the reference express waybill graph q to obtain the same composition G which is most similar to the graph q in the graph GmOr isomorphic diagram gsm(ii) a The calculation of the similar distance specifically includes:
1) nodal texture similarity distance: for node V in graph g and graph qgAnd V, adopting neighbor matching to the detected SIFT feature points, and removing mismatching points according to the distance ratio of the nearest neighbor matching points to the next nearest neighbor to obtain the matching feature point pairs of corresponding nodes in the two graphs
The similarity distance between graph nodes is:
wherein
ωiRepresenting the texture of the graph g and graph q nodes respectively,
logarithm of feature points representing the matching of corresponding nodes, dT(i)∈[0,1]，
Representation node
And viA union of features of (1);
2) node structure similarity distance: the node structure similarity distance comprises corresponding included angles in the graphs g and q and similarity of edges connected with nodes, and is respectively called included angle similarity distance and ray cluster edge similarity distance;
the included angles are similar; in the drawings
And viThe similar distance of the included angle is represented by the cosine distance of the vector:
wherein d isΘ(i)∈[0,1]；
Corresponding side similarity distance: in the two graphs, the corresponding nodes are marked
And viThe ratio of the lengths of all corresponding edges of a connection is X ═ X1，x2，…，xnTherein of
| represents edge eijI represents the current node label, j represents the adjacent point of i, n represents the number of nodes connected with the node i, and the degree of dispersion of X is calculated to obtain the similar distance of all corresponding edges connected by corresponding nodes in the two graphs:
wherein EX and DX are respectively the mean value and variance of variable X; in the drawings
And viThe similar distances of the corresponding sides are:
wherein d isΠ(i)∈[0,1](ii) a According to the above description, the similar distances of the corresponding nodes of the graph g and the reference express waybill graph q are defined as:
d(q(i)，g(i))＝dT(i)+dΘ(i)+dΠ(i) (8)
wherein d (q), (i), g (i)) ∈ [0,3]When similar distance calculation of the reference express waybill graph q and the graph g is carried out, and when corresponding nodes in the graph g and the graph q are absent or wrong in selection, an isomorphic sub-graph g with a smaller similar distance to the graph q is consideredsmThe candidate nodes which are missing or selected wrongly are called outliers, the number of the outliers is represented by outlierNum, and the similar distance of the outliers is calculated in the following mode; similar distance from cluster point:
wherein d isNum∈[0,1]，NqRepresenting the number of nodes V in the reference courier manifest graph q, the similar distances of graph g and graph q are represented as:
wherein c isi∈ {0, 1}, 0 represents that the point is an outlier, 1 represents a node meeting the threshold requirement, and the smaller the value of d (q, G) is, the greater the similarity of the two graphs is, so in the to-be-processed express waybill graph G, the graph G with the minimum similarity distance to the reference express waybill graph q ismOr gsmAnd as a final matching result of the express waybill map G to be processed and the reference express waybill map q:
D(q，G)＝argmin(d(q，g)) (11)
at this time, the final matching result graph gmOr gsmIs recorded as graph gr。
3. The express waybill positioning and extracting method of claim 1, wherein the step 4 specifically comprises:
step e 1: extract graph grCarrying out neighbor matching on SIFT feature points of corresponding nodes in the graph q to obtain a feature point pair set of the corresponding nodes, filtering the set by adopting a random sampling consistency algorithm, and removing mismatching noise point pairs to obtain a node feature pair set; fusing each group of node feature pair set to obtain a matching result graph grAnd the set of pairs of feature points of graph q;
step e 2: adopting a Zhang camera calibration algorithm to establish position mapping transformation of the feature point pairs under different viewing angles according to the feature point pair set of the graph obtained in the step e1, and solving parameters of internal parameters, external parameters and distortion coefficients in the camera calibration algorithm through the corresponding relation of the known feature point pairs to obtain a parameter transformation matrix of the position mapping of the feature point pairs; obtaining a mapping model according to a Zhang calibration camera model, wherein the mapping model comprises the following formula:
the homogeneous coordinate form of the M represents coordinates (u, v,1) under a plane coordinate system, namely coordinates of a reference express waybill map to be processed, and the homogeneous coordinate form of the M represents coordinates (x, y,1) under a world coordinate system, namely coordinates of characteristic points on the reference express waybill map; [ R t]R ═ R as an external parameter of the camera1*r2*r3An image rotation matrix along the direction of XYZ coordinate axes is represented, and t represents a translation matrix; s denotes the scale factor, A denotes the camera intrinsic parameters, A [ R t ]]Namely, the mapping process from the reference express waybill diagram to be processed is shown; for the calibrated two-dimensional plane graph, the world coordinate system Z is 0, so the above formula can be converted into the following form:
wherein H is A [ r ]1r2t]And modifying A, r in H by fitting the characteristic point pair set to H by adopting a maximum likelihood estimation method1、r2T, improvement is obtained
The mapping matrix is a parameter transformation matrix of the mapping of the position of the characteristic point pair, namely a parameter transformation matrix of the mapping from the pattern pixel position on the reference express waybill to the pattern pixel position corresponding to the reference express waybill to be processed;
step e 3: the position of a logistics information filling area manually marked on a reference express waybill is transformed by a parameter transformation matrix
Mapping to the express waybill to be processed to obtain the position of a logistics information filling area corresponding to the express waybill to be processed, and positioning the logistics information filling area filled by a user on the express waybill to be processed;
step e 4: and according to the positioned position and direction of the logistics information filling area filled by the user, performing rotary interpolation calculation on the express waybill to be processed to obtain a horizontal and upright text image, and completing the extraction of the logistics information filled by the user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710880981.7A CN107622247B (en) | 2017-09-26 | 2017-09-26 | Express waybill positioning and extracting method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710880981.7A CN107622247B (en) | 2017-09-26 | 2017-09-26 | Express waybill positioning and extracting method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107622247A CN107622247A (en) | 2018-01-23 |
| CN107622247B true CN107622247B (en) | 2020-08-25 |
Family
ID=61090798
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710880981.7A Active CN107622247B (en) | 2017-09-26 | 2017-09-26 | Express waybill positioning and extracting method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107622247B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108509870B (en) * | 2018-03-14 | 2019-07-12 | 安徽工大信息技术有限公司 | A kind of Eriocheir sinensis uniqueness recognition methods based on images match |
| CN109049997A (en) * | 2018-07-26 | 2018-12-21 | 张亮 | A kind of novel express delivery sorting system |
| CN109190629A (en) * | 2018-08-28 | 2019-01-11 | 传化智联股份有限公司 | A kind of electronics waybill generation method and device |
| CN110210488A (en) * | 2019-06-14 | 2019-09-06 | 上海中通吉网络技术有限公司 | The recognition methods of bar code and cell-phone number and device on a kind of express waybill |
| CN112308915A (en) * | 2020-03-30 | 2021-02-02 | 北京沃东天骏信息技术有限公司 | Method and device for positioning express package |
| CN111553317B (en) * | 2020-05-14 | 2023-08-08 | 北京惠朗时代科技有限公司 | Anti-fake code acquisition method and device, computer equipment and storage medium |
| CN113762846B (en) * | 2020-10-22 | 2024-04-16 | 北京京东振世信息技术有限公司 | Method and device for distinguishing face sheet text |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102289681A (en) * | 2011-08-05 | 2011-12-21 | 上海邮政科学研究院 | Method for matching envelope images |
| CN105354550A (en) * | 2015-11-03 | 2016-02-24 | 华东师范大学 | Form content extracting method based on registration of local feature points of image |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW501035B (en) * | 2001-03-20 | 2002-09-01 | Ulead Systems Inc | Interactive image searching method based on local object |
-
2017
- 2017-09-26 CN CN201710880981.7A patent/CN107622247B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102289681A (en) * | 2011-08-05 | 2011-12-21 | 上海邮政科学研究院 | Method for matching envelope images |
| CN105354550A (en) * | 2015-11-03 | 2016-02-24 | 华东师范大学 | Form content extracting method based on registration of local feature points of image |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107622247A (en) | 2018-01-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107622247B (en) | Express waybill positioning and extracting method | |
| US9245345B2 (en) | Device for generating three dimensional feature data, method for generating three-dimensional feature data, and recording medium on which program for generating three-dimensional feature data is recorded | |
| CN107424142B (en) | Weld joint identification method based on image significance detection | |
| CN110363158A (en) | A kind of millimetre-wave radar neural network based cooperates with object detection and recognition method with vision | |
| CN105046252B (en) | A kind of RMB prefix code recognition methods | |
| CN101520849B (en) | Reality augmenting method and reality augmenting system based on image characteristic point extraction and random tree classification | |
| CN106971406B (en) | Object pose detection method and device | |
| CN105426825B (en) | A kind of power grid geographical wiring diagram method for drafting based on Aerial Images identification | |
| CN110321769A (en) | A kind of more size commodity on shelf detection methods | |
| CN109461132B (en) | SAR image automatic registration method based on feature point geometric topological relation | |
| CN112598627A (en) | Method, system, electronic device and medium for detecting image defects | |
| CN107689050A (en) | A kind of depth image top sampling method based on Color Image Edge guiding | |
| JP4946878B2 (en) | Image identification apparatus and program | |
| CN109767431A (en) | Accessory appearance defect inspection method, device, equipment and readable storage medium storing program for executing | |
| CN113642463B (en) | Heaven and earth multi-view alignment method for video monitoring and remote sensing images | |
| CN108830224A (en) | A kind of high-resolution remote sensing image Ship Target Detection method based on deep learning | |
| CN115063802A (en) | PSENet-based circular seal identification method, device and medium | |
| CN108805139A (en) | A kind of image similarity computational methods based on frequency-domain visual significance analysis | |
| CN102831419B (en) | Method for detecting and blurring plate number in street view image rapidly | |
| CN112949755B (en) | OCR data synthesis method based on image structure information | |
| CN107358244B (en) | A kind of quick local invariant feature extracts and description method | |
| CN112184533A (en) | Watermark synchronization method based on SIFT feature point matching | |
| CN107564015A (en) | A kind of segmentation and labeling method and device based on polychrome level image connectivity domain | |
| Chiang et al. | Extracting road vector data from raster maps | |
| CN115512138A (en) | Line feature matching method based on point-line-plane fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |