CN110956088B - Overlapped text line positioning and segmentation method and system based on deep learning - Google Patents
Overlapped text line positioning and segmentation method and system based on deep learning Download PDFInfo
- Publication number
- CN110956088B CN110956088B CN201911053860.0A CN201911053860A CN110956088B CN 110956088 B CN110956088 B CN 110956088B CN 201911053860 A CN201911053860 A CN 201911053860A CN 110956088 B CN110956088 B CN 110956088B
- Authority
- CN
- China
- Prior art keywords
- text line
- overlapping
- overlapping text
- line region
- line area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/40—Document-oriented image-based pattern recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/28—Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/14—Image acquisition
- G06V30/148—Segmentation of character regions
- G06V30/153—Segmentation of character regions using recognition of characters or words
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Artificial Intelligence (AREA)
- Character Input (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an overlapping text line positioning and dividing method and system based on deep learning, and belongs to the field of computer vision. The method comprises the following steps: preprocessing an original image; inputting the non-overlapping text line region feature score map, the overlapping text line region feature score map and the link information feature score map among text line region pixels into a trained instance segmentation full convolution neural network; acquiring outlines of a non-overlapping text line area and an overlapping text line area; merging the non-overlapping text line areas into the overlapping text line areas; and performing quadrilateral fitting on the text line area to obtain an external quadrilateral of the text line area, and realizing positioning segmentation of overlapped text lines. The method can effectively solve the difficult problem of positioning and dividing the overlapped text lines, and can complete the task which cannot be completed by the traditional method. In addition, the method can achieve good algorithm performance by only needing less training data and training iteration rounds and matching with simple post-processing.
Description
Technical Field
The invention relates to the field of computer vision, in particular to an overlapping text line positioning and dividing method and system based on deep learning.
Background
In many application scenarios, there is a need to electronically record document picture content to generate structured data and complete automated recording. Such a need can be addressed using OCR (Optical Character Recognition) techniques. In general, OCR technology includes two major steps of text detection (text detection) and text recognition (text recognition). Conventional text detection methods typically employ connected-domain analysis (Connected Component Analysis, CCA) or a Sliding Window detection mechanism (SW). These methods typically require manual design of a series of rules to extract low-level or medium-level features from the image, and combine complex pre-and post-processing procedures to accomplish the text detection task. Limited feature representation capability and complex processing flow of manual design rules, the traditional methods are difficult to perform with high price, and particularly in difficult recognition scenes such as fuzzy characters, overlapped characters, scene characters with complex background and the like.
In recent years, deep learning techniques have rapidly evolved and have been successfully applied to text detection and recognition tasks. In essence, deep learning is an algorithm of feature learning that approximates a potential function mapping from input to output by automatically learning and extracting features of the input object (image, text, etc.), fitting a specific target output label. A deep learning model is typically composed of a series of sequential operations that must be differentiable so that optimization methods such as gradient descent can be employed to achieve end-to-end training optimization.
Although deep learning techniques have greatly improved the performance of document text detection algorithms, and even harder scene text detection tasks, it has to be acknowledged that some special harder text detection tasks, such as overlapping text line detection, remain quite challenging. As shown in fig. 1. Such overlapping lines of text are present in large numbers in tickets, forms, and pictures of documents, and are typically caused by offset, tilt, and even nested printing. If the problems of detection and identification of the text can be well solved, the performance of structured input of the object can be greatly improved, so that the method has great practical application value.
Disclosure of Invention
The invention relates to a positioning and segmentation method of overlapped text lines based on deep learning, which can well solve the detection problem of the overlapped text lines in various notes, forms and document images shot by scanners, high-speed cameras and mobile phones, provide more accurate text line area information for subsequent recognition tasks so as to improve the overall recognition precision and further finish the automatic input work of structured data with high quality.
According to a first aspect of the present invention, there is provided a method for positioning and dividing overlapping text lines based on deep learning, the method comprising the steps of:
step 2, training the example segmentation full convolutional neural network, inputting the preprocessed original image into the trained example segmentation full convolutional neural network, and outputting a non-overlapping text line region feature score map, an overlapping text line region feature score map and a link information feature score map among pixels of the text line region;
step 3, acquiring the outlines of the non-overlapping text line area and the overlapping text line area based on the non-overlapping text line area characteristic score map, the overlapping text line area characteristic score map and the link information characteristic score map among pixels of the text line area by a connected domain analysis method;
step 4, merging the non-overlapping text line area to the overlapping text line area according to the outline of the non-overlapping text line area and the outline of the overlapping text line area;
and 5, performing quadrilateral fitting on the combined text line areas to obtain circumscribed quadrilaterals of the text line areas, and realizing positioning segmentation of overlapped text lines.
Further, the step 1 specifically includes: and carrying out boundary filling N units on the input original image, and then carrying out 1/M downsampling to obtain the preprocessed original image, wherein M and N are integers which are equal to or larger than 1, and M is an integer multiple of N.
Further, the step 2 specifically includes:
step 21: labeling each sample image in the training sample set through the outline of the text line area represented by the quadrangle, and generating a labeled tag file;
step 22: the method comprises the steps of sending a label file and a sample image into an instance segmentation full-convolution instance segmentation network for training, wherein in order to complete supervision and learning of overlapped text lines, the instance segmentation full-convolution instance segmentation network automatically calculates the outline of an overlapped text line area according to the outline of a Chinese text line area in the label file, takes the outline as a supervision and learning target of the overlapped text line area, and combines the outline of a non-overlapped text line area to complete a training process to form a preliminary training model;
step 23: aiming at the preliminary training model, testing is carried out through a test sample set, the detection segmentation precision of the non-overlapping text line region and the overlapping text line region is evaluated, if the precision requirement is met, the training process is terminated, and the preliminary training model is used as a trained example to segment the full convolution neural network; if the accuracy requirement is not met, increasing the training sample size, adjusting the structure of the instance segmentation full convolution instance segmentation network and training parameters, and repeating the training process until a trained instance segmentation full convolution neural network meeting the accuracy requirement is obtained;
step 24: and inputting the preprocessed original image into a trained instance segmentation full convolution neural network, and outputting a non-overlapping text line region feature score map, an overlapping text line region feature score map and a link information feature score map among text line region pixels.
Further, the step 3 specifically includes:
step 31: setting a first threshold for the non-overlapping text line region feature score map, a second threshold for the overlapping text line region feature score map, and a third threshold for the link information feature score map between text line region pixels;
step 32: binarization processing is carried out on the non-overlapping text line area characteristic score diagram according to a first threshold value, binarization processing is carried out on the overlapping text line area characteristic score diagram according to a second threshold value, binarization processing is carried out on the link information characteristic score diagram between text line area pixels according to a third threshold value, non-overlapping text line area pixel points and background pixel points are obtained in the non-overlapping text line area characteristic score diagram, overlapping text line area pixel points and background pixel points are obtained in the overlapping text line area characteristic score diagram, and link state information and non-link state information are obtained in the link information characteristic score diagram between text line area pixels;
step 33: and obtaining a pixel point region of the non-overlapping text line region according to the pixel point combination link state information of the non-overlapping text line region, obtaining a pixel point region of the overlapping text line region according to the pixel point combination link state information of the overlapping text line region, and representing the outline of the pixel point region by using a connected domain.
Further, the value ranges of the first threshold, the second threshold and the third threshold are all [0,1].
Further, the step 4 specifically includes:
step 41: combining the pixel point area of the non-overlapping text line area and the pixel point area of the overlapping text line area;
step 42: judging adjacent information between adjacent pixel points, combining a characteristic score graph of link information between pixels of a text line region, and merging two pixel points into a connected domain when the two pixel points are adjacent and the link information of the two pixel points is positive;
further, two pixels are adjacent to each other: the two pixel points differ by 1-3 pixels in the X-direction pixel coordinate axis or the Y-direction pixel coordinate axis.
Step 43: and acquiring an optimal distance threshold value on a variable distance threshold value test set by adopting a strategy based on variable distance threshold value combination and adopting a dynamic searching distance threshold value mode based on end-to-end detection precision, and carrying out combination operation if the distance between two connected domains is within the optimal distance threshold value range.
According to a second aspect of the present invention, there is provided a deep learning based overlapping text line location splitting device, comprising:
an original image input section for inputting an original image containing overlapping text lines, the original image being preprocessed;
the feature score map output component is used for inputting the preprocessed original image into the trained instance segmentation full convolution neural network and outputting a non-overlapping text line region feature score map, an overlapping text line region feature score map and a link information feature score map among text line region pixels;
contour obtaining means for obtaining, by a connected domain analysis method, contours of the non-overlapping text line region and the overlapping text line region based on the non-overlapping text line region feature score map, the overlapping text line region feature score map, and the link information feature score map between pixels of the text line region;
a region merging section for merging the non-overlapping text line region to the overlapping text line region based on the outline of the non-overlapping text line region and the overlapping text line region;
and the result output part is used for performing quadrilateral fitting on the text line area to obtain an external quadrilateral of the text line area, so as to realize positioning segmentation of overlapped text lines.
According to a third aspect of the present invention, there is provided a deep learning based overlapping text line localization segmentation system, the system comprising:
a processor and a memory for storing executable instructions;
wherein the processor is configured to execute the executable instructions to perform the deep learning based overlapping text line location segmentation method as set forth in any of the preceding aspects.
According to a fourth aspect of the present invention, there is provided a computer-readable storage medium, having stored thereon a computer program,
the computer program, when executed by a processor, implements the deep learning based overlapping text line localization segmentation method as described in any of the previous aspects.
The invention has the beneficial effects that:
1. image features are automatically extracted and learned based on a deep learning full convolution network instance segmentation method, so that difficult manual rule design and complex preprocessing and post-processing flows are avoided;
2. the method can adapt to different types of document images and different types of text line overlapping patterns, and solves the problem that the traditional method cannot solve;
3. the designed full convolution network finally outputs Score Maps, namely Score Maps, representing the predicted confidence level, and the confidence level can effectively guide the subsequent recognition and even structuring work;
4. the method is simple and efficient, the whole flow is composed of the FCN network and simple and efficient post-processing logic, and the requirements of practical application are met.
5. The labeling training process is simple, and the training process can be efficiently completed on the premise that the overlapped area does not need to be specially labeled.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to the structures shown in these drawings without inventive effort for a person skilled in the art.
FIG. 1 illustrates a typical prior art overlapping text market;
FIG. 2 shows a flow chart of an overlapping text line location segmentation method based on deep learning according to the present invention;
FIG. 3 illustrates an example of overlapping text lines in a deep learning based overlapping text line localization segmentation method in accordance with the present invention;
FIG. 4 illustrates a non-overlapping text line region feature score plot in a deep learning based overlapping text line location segmentation method in accordance with the present invention;
FIG. 5 illustrates an overlapping text line region feature score plot in a deep learning based overlapping text line location segmentation method in accordance with the present invention;
FIG. 6 shows a schematic diagram of an optimal threshold search process in a deep learning based overlapping text line location segmentation method according to the present invention;
FIG. 7 illustrates an exemplary circumscribed quadrilateral schematic of overlapping text lines in a deep learning based overlapping text line location segmentation method in accordance with the present invention;
fig. 8 shows an effect diagram of an overlapping text line location segmentation method based on deep learning according to the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.
The terms first, second and the like in the description and in the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the disclosure described herein may be capable of operation in sequences other than those illustrated or described herein, for example.
Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
A plurality, including two or more.
And/or, it should be understood that for the term "and/or" used in this disclosure, it is merely one association relationship describing associated objects, meaning that there may be three relationships. For example, a and/or B may represent: a exists alone, A and B exist together, and B exists alone.
The invention relates to a deep learning-based overlapped text line positioning and segmentation method, which comprises the following steps:
inputting an original image containing overlapped text lines, and preprocessing the original image;
inputting the preprocessed original image into a trained instance segmentation full convolution neural network, and outputting a non-overlapping text line region feature score map, an overlapping text line region feature score map and a link information feature score map among pixels of the text line region;
acquiring outlines of a non-overlapping text line region and an overlapping text line region based on the non-overlapping text line region feature score map, the overlapping text line region feature score map and the link information feature score map among pixels of the text line region by a connected domain analysis method;
merging the non-overlapping text line region into the overlapping text line region according to the outline of the non-overlapping text line region and the overlapping text line region;
and performing quadrilateral fitting on the text line area to obtain an external quadrilateral of the text line area, and realizing positioning segmentation of overlapped text lines.
Examples
The invention relates to an accurate positioning and dividing method for overlapped text lines. Aiming at the difficult problem of overlapping text line detection in bills, forms and document images, the team innovatively adopts an example segmentation full convolution neural network model, integrates the detection of text regions and segmentation tasks of different text line example regions into a neural network, adopts an innovative post-processing method, and generates complete and accurate text line example outlines by combining a text line segmentation diagram and an overlapping region segmentation diagram, thereby accurately determining different text line region coordinates.
The specific flow is shown in fig. 2:
the first step of image preprocessing: the input image is preprocessed, it is important that the boundaries are padded so that the width and height of the image are not affected by the downsampling, and the value of the aligned boundaries is generally consistent with the downsampled value. For example, where the downsampling is 1/16, the boundary alignment is 16 units or pixels, or an integer multiple of 16, such as 32,64, etc.
Second step of text instance segmentation at pixel level: and sending the preprocessed image into a trained instance segmentation full convolution neural network, and outputting a plurality of feature Score Maps (Score Maps) which respectively represent the background, the text region, the overlapped text region and the feature map of link information among pixels of the text region.
Training process for example segmentation full convolution network: firstly, labeling each sample in a sample set, wherein the labeling content mainly comprises the outline of a text line region represented by a quadrangle, and special labeling is not required for the overlapped text line region, so that a label file is generated. The file and the image file are sent together to a full convolution instance segmentation network for training. In order to complete supervision and learning of the overlapped text line areas, outline information of the overlapped areas is automatically calculated according to outlines of the text line examples in the label file in network preprocessing, and the outline information is sequentially used as a supervision target of the overlapped areas and combined with a target of a non-overlapped area to jointly complete a training task. When one round of training is completed, the accuracy of the whole text line detection segmentation needs to be evaluated on the test set for the trained model, wherein the accuracy evaluation of the overlapping area is also included. If the expected effect and the accuracy index are reached, the training process can be terminated, and the model is used as prediction; if not, the training sample size is increased, and the structure, training parameters and the like of the full convolution example segmentation network model are possibly adjusted, and the training process is repeated until the estimated performance meets the requirement. The possible adjustment of the model structure is usually from two aspects, namely, the adjustment of the model capacity on one hand, and the aim is to improve the characteristic learning capacity of the model, including the number of layers of the convolutional neural network, the number of filters of the convolutional operation of each layer, the characteristic map fusion mode, the style of the nonlinear activation function and the like; another aspect is the adjustment of model generalization ability, such as adjustment of regularization term parameters in the network, with the goal of improving the performance of the model on the test set (i.e., the unworn samples). The possible adjustment of training parameters generally comprises several aspects, namely, adjustment of super parameters of the training process, such as adjustment of learning rate attenuation strategy and initial size, adjustment of training batch size and overall iteration number, and the like; on the other hand, the adjustment of the training loss function comprises the adjustment of the loss function style and the super parameters related to the loss function, and the like.
Taking fig. 3 as an example, a non-overlapping text line area feature score map and an overlapping text line area feature score map are described below.
1) The non-overlapping text line area Score Map, as shown in fig. 4, has each pixel value representing the confidence that the pixel is inside the text line area, normalized to the [0,1] interval.
2) The overlapping text line area Score Map, as shown in fig. 5, each pixel value represents the confidence that the pixel is in the overlapping text area, again normalized to the [0,1] interval.
Thirdly, extracting text region and overlapping region outlines: based on the connected domain analysis method, the contour information of the non-overlapping text line region and the overlapping text line region is obtained by creatively combining the non-overlapping text line region, the overlapping text line region Score maps and the Score maps of the inter-pixel link information in the text region.
Fourth step, merging the overlapped text line area outline to the non-overlapped text line area outline: the step is to complete the merging of the outline of each text line instance and the outline of the overlapped area, and the process adopts the methods of connected domain analysis and multi-connected domain merging to generate the complete outline of each text line area. The connected domain analysis process creatively combines the adjacent position information between pixels and the link information between adjacent pixels predicted by the full convolution neural network, namely, only when two pixels are adjacent and the link information of the predicted pixels is positive, the two pixels are considered to be combined into one connected domain. In the multi-connected domain merging process, a strategy based on variable distance threshold merging is innovatively adopted, and an optimal distance threshold is obtained on a test set in a dynamic searching distance threshold mode based on end-to-end detection precision. When the optimal threshold is determined, the merging operation is performed as long as the distance between the communicating domains is within the threshold range. The best threshold search process is shown in fig. 6, and specifically includes:
and testing the full convolution instance segmentation network through a variable distance threshold test set, setting a distance threshold search range interval, such as [0,5], and carrying out optimal distance threshold search to obtain an optimal distance threshold.
Setting a distance threshold search range interval to perform optimal distance threshold search comprises:
traversing a threshold interval, wherein the step length is 1; applying the threshold to perform text line contour merging; calculating the overall detection segmentation precision of the test set; and storing the maximum precision and the optimal threshold, and taking the threshold as an optimal distance threshold.
Experiments have shown that in combination with these two innovative approaches, the algorithm exhibits a relatively high degree of accuracy in overlapping text contours and text contour combinations.
And fifthly, performing quadrilateral fitting on the text line instance area to obtain an external quadrilateral of the text line. As shown in fig. 7.
Fig. 8 shows some effect diagrams of overlapping text line location segmentation using the technical solution of the present invention. Experiments show that the method can very effectively solve the difficult problem of positioning and dividing the overlapped text lines, and can complete tasks which cannot be completed by the traditional method. In addition, the method can achieve good algorithm performance by only matching less training data with training iteration rounds and simple post-processing, and therefore has great practical application value.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be apparent to those skilled in the art that the above implementation may be implemented by means of software plus necessary general purpose hardware platform, or of course by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The embodiments of the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present invention and the scope of the claims, which are to be protected by the present invention.
Claims (7)
1. The overlapping text line positioning and dividing method based on deep learning is characterized by comprising the following steps of:
step 1, inputting an original image containing overlapped text lines, and preprocessing the original image;
step 2, training the example segmentation full convolutional neural network, inputting the preprocessed original image into the trained example segmentation full convolutional neural network, and outputting a non-overlapping text line region feature score map, an overlapping text line region feature score map and a link information feature score map among pixels of the text line region;
step 3, acquiring the outlines of the non-overlapping text line area and the overlapping text line area based on the non-overlapping text line area characteristic score map, the overlapping text line area characteristic score map and the link information characteristic score map among pixels of the text line area by a connected domain analysis method;
step 4, merging the non-overlapping text line area to the overlapping text line area according to the outline of the non-overlapping text line area and the outline of the overlapping text line area;
step 5, performing quadrilateral fitting on the combined text line area to obtain the circumscribed quadrilateral of the text line area, realizing the positioning segmentation of overlapped text lines,
wherein, the step 3 specifically includes:
step 31: setting a first threshold for the non-overlapping text line region feature score map, a second threshold for the overlapping text line region feature score map, and a third threshold for the link information feature score map between text line region pixels;
step 32: binarization processing is carried out on the non-overlapping text line area characteristic score diagram according to a first threshold value, binarization processing is carried out on the overlapping text line area characteristic score diagram according to a second threshold value, binarization processing is carried out on the link information characteristic score diagram between text line area pixels according to a third threshold value, non-overlapping text line area pixel points and background pixel points are obtained in the non-overlapping text line area characteristic score diagram, overlapping text line area pixel points and background pixel points are obtained in the overlapping text line area characteristic score diagram, and link state information and non-link state information are obtained in the link information characteristic score diagram between text line area pixels;
step 33: obtaining a pixel point region of the non-overlapping text line region according to the pixel point combination link state information of the non-overlapping text line region, obtaining a pixel point region of the overlapping text line region according to the pixel point combination link state information of the overlapping text line region, and representing the outline of the pixel point region by using a connected domain;
wherein, the step 4 specifically includes:
step 41: combining the pixel point area of the non-overlapping text line area and the pixel point area of the overlapping text line area;
step 42: judging adjacent information between adjacent pixels, combining a characteristic score graph of link information between pixels of a text line region, and merging two pixels into a connected domain when the two pixels are adjacent and the link state information of the two pixels is positive; wherein, two adjacent pixels refer to: the two pixel points are different by 1-3 pixels in the X-direction pixel coordinate axis or the Y-direction pixel coordinate axis;
step 43: and acquiring an optimal distance threshold value on a variable distance threshold value test set by adopting a strategy based on variable distance threshold value combination and adopting a dynamic distance threshold searching mode based on end-to-end detection precision, and carrying out combination operation if the distance between two connected domains is within the optimal distance threshold value range.
2. The method for positioning and splitting overlapped text lines based on deep learning according to claim 1, wherein the step 1 specifically includes: and carrying out boundary filling N units on the input original image, and then carrying out 1/M downsampling to obtain the preprocessed original image, wherein M and N are integers more than or equal to 1, and M is integer multiple of N.
3. The method for positioning and splitting overlapped text lines based on deep learning according to claim 1, wherein the step 2 specifically comprises:
step 21: labeling each sample image in the training sample set through the outline of the text line area represented by the quadrangle, and generating a labeled tag file;
step 22: the method comprises the steps of sending a label file and a sample image into an example segmentation full convolution neural network for training, wherein in order to complete supervision and learning of overlapped text lines, the example segmentation full convolution neural network automatically calculates the outline of an overlapped text line area according to the outline of a text line area in the label file, takes the outline as a supervision and learning target of the overlapped text line area, and combines the outline of a non-overlapped text line area to complete a training process to form a preliminary training model;
step 23: aiming at the preliminary training model, testing is carried out through a test sample set, the detection segmentation precision of the non-overlapping text line region and the overlapping text line region is evaluated, if the precision requirement is met, the training process is terminated, and the preliminary training model is used as a trained example to segment the full convolution neural network; if the accuracy requirement is not met, increasing the training sample size, adjusting the structure and training parameters of the instance segmentation full convolution neural network, and repeating the training process until the trained instance segmentation full convolution neural network meeting the accuracy requirement is obtained;
step 24: and inputting the preprocessed original image into a trained instance segmentation full convolution neural network, and outputting a non-overlapping text line region feature score map, an overlapping text line region feature score map and a link information feature score map among text line region pixels.
4. The method for positioning and partitioning overlapped text lines based on deep learning as set forth in claim 1, wherein the first threshold, the second threshold and the third threshold are all in the value ranges of [0,1].
5. An overlapped text line location splitting device based on deep learning, characterized in that the overlapped text line location splitting device operates based on the overlapped text line location splitting method according to any one of claims 1 to 4, the overlapped text line location splitting device comprising:
an original image input section for inputting an original image containing overlapping text lines, the original image being preprocessed;
the feature score map output component is used for inputting the preprocessed original image into the trained instance segmentation full convolution neural network and outputting a non-overlapping text line region feature score map, an overlapping text line region feature score map and a link information feature score map among text line region pixels;
contour obtaining means for obtaining, by a connected domain analysis method, contours of the non-overlapping text line region and the overlapping text line region based on the non-overlapping text line region feature score map, the overlapping text line region feature score map, and the link information feature score map between pixels of the text line region;
a region merging section for merging the non-overlapping text line region to the overlapping text line region based on the outline of the non-overlapping text line region and the overlapping text line region;
and the result output part is used for performing quadrilateral fitting on the text line area to obtain an external quadrilateral of the text line area, so as to realize positioning segmentation of overlapped text lines.
6. An overlapping text line location segmentation system based on deep learning, the system comprising:
a processor and a memory for storing executable instructions;
wherein the processor is configured to execute the executable instructions to perform the deep learning based overlapping text line location segmentation method of any one of claims 1 to 4.
7. A computer-readable storage medium, having a computer program stored thereon,
the computer program, when executed by a processor, implements the deep learning based overlapping text line localization segmentation method as claimed in any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911053860.0A CN110956088B (en) | 2019-10-31 | 2019-10-31 | Overlapped text line positioning and segmentation method and system based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911053860.0A CN110956088B (en) | 2019-10-31 | 2019-10-31 | Overlapped text line positioning and segmentation method and system based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110956088A CN110956088A (en) | 2020-04-03 |
| CN110956088B true CN110956088B (en) | 2023-06-30 |
Family
ID=69976607
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911053860.0A Active CN110956088B (en) | 2019-10-31 | 2019-10-31 | Overlapped text line positioning and segmentation method and system based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110956088B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111539269A (en) * | 2020-04-07 | 2020-08-14 | 北京达佳互联信息技术有限公司 | Text region identification method and device, electronic equipment and storage medium |
| CN112232336A (en) * | 2020-09-02 | 2021-01-15 | 深圳前海微众银行股份有限公司 | Certificate identification method, device, equipment and storage medium |
| CN114419641B (en) * | 2022-03-15 | 2022-06-21 | 腾讯科技（深圳）有限公司 | Training method and device of text separation model, electronic equipment and storage medium |
| CN116152842B (en) * | 2022-11-18 | 2023-11-03 | 北京中卡信安电子设备有限公司 | Certificate image processing method and device, storage medium and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108549893A (en) * | 2018-04-04 | 2018-09-18 | 华中科技大学 | A kind of end-to-end recognition methods of the scene text of arbitrary shape |
| CN109902622A (en) * | 2019-02-26 | 2019-06-18 | 中国科学院重庆绿色智能技术研究院 | A kind of text detection recognition methods for boarding pass information verifying |
| CN109919025A (en) * | 2019-01-30 | 2019-06-21 | 华南理工大学 | Video scene Method for text detection, system, equipment and medium based on deep learning |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9430703B2 (en) * | 2014-12-19 | 2016-08-30 | Konica Minolta Laboratory U.S.A., Inc. | Method for segmenting text words in document images using vertical projections of center zones of characters |
-
2019
- 2019-10-31 CN CN201911053860.0A patent/CN110956088B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108549893A (en) * | 2018-04-04 | 2018-09-18 | 华中科技大学 | A kind of end-to-end recognition methods of the scene text of arbitrary shape |
| CN109919025A (en) * | 2019-01-30 | 2019-06-21 | 华南理工大学 | Video scene Method for text detection, system, equipment and medium based on deep learning |
| CN109902622A (en) * | 2019-02-26 | 2019-06-18 | 中国科学院重庆绿色智能技术研究院 | A kind of text detection recognition methods for boarding pass information verifying |
Non-Patent Citations (2)
| Title |
|---|
| 基于YOLO的自然场景倾斜文本定位方法研究;周翔宇等;《计算机工程与应用》(第09期);全文 * |
| 基于语义分割技术的任意方向文字识别;王涛等;《应用科技》;20170704(第03期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110956088A (en) | 2020-04-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110956088B (en) | Overlapped text line positioning and segmentation method and system based on deep learning | |
| EP1693782B1 (en) | Method for facial features detection | |
| US11475681B2 (en) | Image processing method, apparatus, electronic device and computer readable storage medium | |
| CN108416902B (en) | Real-time object identification method and device based on difference identification | |
| CN110647829A (en) | Bill text recognition method and system | |
| CN109118473B (en) | Angular point detection method based on neural network, storage medium and image processing system | |
| CN110866871A (en) | Text image correction method and device, computer equipment and storage medium | |
| CN111259889A (en) | Image text recognition method and device, computer equipment and computer storage medium | |
| CN110569878A (en) | Photograph background similarity clustering method based on convolutional neural network and computer | |
| CN110287952B (en) | Method and system for recognizing characters of dimension picture | |
| CN113221925B (en) | Target detection method and device based on multi-scale image | |
| KR101997048B1 (en) | Method for recognizing distant multiple codes for logistics management and code recognizing apparatus using the same | |
| CN112101195B (en) | Crowd density estimation method, crowd density estimation device, computer equipment and storage medium | |
| CN108830175A (en) | Iris image local enhancement methods, device, equipment and storage medium | |
| Xing et al. | Traffic sign recognition using guided image filtering | |
| US11367206B2 (en) | Edge-guided ranking loss for monocular depth prediction | |
| CN111914596B (en) | Lane line detection method, device, system and storage medium | |
| CN114913338A (en) | Segmentation model training method and device, and image recognition method and device | |
| Aliev et al. | Algorithm for choosing the best frame in a video stream in the task of identity document recognition | |
| CN113989604A (en) | Tire DOT information identification method based on end-to-end deep learning | |
| CN112365451A (en) | Method, device and equipment for determining image quality grade and computer readable medium | |
| CN116798041A (en) | Image recognition method and device and electronic equipment | |
| JP7246104B2 (en) | License plate identification method based on text line identification | |
| Huang et al. | FAST and FLANN for feature matching based on SURF | |
| CN115953744A (en) | Vehicle identification tracking method based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100083 office A-501, 5th floor, building 2, yard 1, Nongda South Road, Haidian District, Beijing Applicant after: BEIJING YIDAO BOSHI TECHNOLOGY Co.,Ltd. Address before: 100083 office a-701-1, a-701-2, a-701-3, a-701-4, a-701-5, 7th floor, building 2, No.1 courtyard, Nongda South Road, Haidian District, Beijing Applicant before: BEIJING YIDAO BOSHI TECHNOLOGY Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Wang Yong Inventor after: Zhu Junmin Inventor after: Kang Tiegang Inventor after: Shi Wei Inventor before: Wang Yong Inventor before: Zhu Junmin Inventor before: Kang Tiegang Inventor before: Shi Wei |
|
| CB03 | Change of inventor or designer information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |