JP2014229314A - Method and device for text detection - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and device for text detection in a natural image and a method and system for text information extraction.SOLUTION: A method for detecting a text region in an image including at least one connected component (CC) includes: a TBR detection step 310 for detecting at least one text background region (TBR) from an image; a CC filtering step 320 for ensuring at least one candidate text CC by filtering at least one CC; and a CC grouping step 330 for forming at least one CC group by grouping at least one candidate text CC on the basis of the TBR detected in the TBR detection step and generating at least one text region on the basis of at least one CC group. A location of a text in an image can be promptly and/or accurately shown, and as a result, performance in text detection increases.

Description

  The present invention relates to text detection, and more particularly, to a text detection method and apparatus, and a text information extraction method and system.

  Detecting text in images, especially natural images, is useful for many image recognition applications such as computerized assistance for visually impaired and foreigners, automatic image and video search, and robot navigation in urban environments. Very important.

  Nevertheless, text detection in natural scenes is a difficult problem. In contrast to scanning printed pages, faxes and business cards, the main challenge is text diversity such as font, size, skew angle, italics and skew distortion. It is more complicated by environmental factors such as uneven illumination and reflection, poor lighting conditions, and complex backgrounds.

  In related literature, a text detection method for detecting a text region in a natural scene usually follows the flowchart shown in FIG. The method 100 of FIG. 1 begins at block 110 and generates components from an image. Here, the component may be a connected component (CC) that is a pixel group having a similar color, gray scale, or stroke width.

  Next, at block 120, various features are extracted from each component, and the non-text components are filtered based on the features to ensure candidate text components.

  Next, at block 130, the reserved candidate text components are grouped together to form a text line or word, and as a text line or word bounding box (minimum polygon such as a rectangle containing text). Output text area.

  A general problem with the prior art is that the prior art attempts to detect text only by text region features such as edges, corners, strokes, colors and textures. However, context information around text that can be useful in most cases is ignored. As a result, in the prior art, false detection due to complex non-text regions and non-detection due to wide variety of text in natural scenes occur. Therefore, there are severe tradeoffs when trying to detect text directly only by the characteristics of the text region.

  Therefore, there is a need for an improved method for text detection in images that utilize background information around the text region.

  ‘Scenario charactor’, published in the text of the ‘Nenhichi Kunshige’, ‘Fen Yakkai’, ‘Senichi Uchida’, ‘International Conferencing on Document’, and the ‘Nen. A detection method is proposed. In particular, the literature extracts context features from the expanded region of the target component formed by adding a 10 pixel margin around the target component. The method further classifies the components into one of six scene component categories: “sky”, “green”, “signboard”, “ground”, “building” and “other”. One problem with this method is that general categories such as “sky” and “green” are inappropriate for text detection, and “signboards” are all types of logos, tags, scrolls, posters, etc. The text background area is not included in the range. Another problem with this method is that the context information is extracted from a fixed size blank area that cannot adapt to scene variations. Yet another problem is that while the information is useful in CC grouping, this method cannot obtain CC relationships in space, for example which CC belongs to a signboard.

  Therefore, at least one of the above problems needs to be addressed.

  The inventor has noted that most text in a natural scene is printed on a relatively similar background area with high contrast to the text so that it can be easily recognized. This may be useful for text detection.

  Thereby, a novel text detection method and apparatus is proposed in the present invention to improve the performance of text detection in images, especially natural scene images. In order to define this general background area surrounding the text, a new concept, text background area (TBR), is introduced into the present invention. In a natural scene image, TBR usually exists as a signboard, logo, tag, scroll, poster, or the like, but is not limited to these forms. Rather than finding the text region directly, the text is searched by first finding the TBR and classifying the component as a component within the TBR and out of all TBRs, ie, components of the outer region (OR). It is assumed that the text is more likely to appear in the TBR than outside the TBR. Also, a text line / word rarely intersects two regions (two TBRs or TBR and OR). Based on that assumption, the TBR information may be used in connected component filtering and / or connected component grouping.

  According to a first aspect of the invention, there is provided a method for detecting a text region in an image comprising at least one connected component (CC). The method includes: a TBR detection step for detecting at least one text background region (TBR) from the image; a CC filtering step for filtering at least one CC to ensure at least one candidate text CC; and a TBR detection step. A CC grouping step for grouping at least one candidate text CC based on the detected TBR to form at least one CC group and generating at least one text region based on the at least one CC group; Is provided.

  According to a second aspect of the present invention, there is provided a text detection device for detecting a text region in an image including at least one connected component (CC). The apparatus includes a TBR detection unit configured to detect a text background region (TBR) from the image, and CC filtering configured to filter at least one CC to ensure at least one candidate text CC. A unit and at least one candidate text CC based on the TBR detected in the TBR detection unit to form at least one CC group, and at least one text region based on the at least one CC group. A CC grouping unit configured to generate.

  According to a third aspect of the present invention, a text information extraction method is provided. The method includes detecting a text region from an input image or input video using the text detection method according to the first aspect of the present invention, extracting text from the detected text region, and extracted text Recognizing and acquiring text information.

  According to a fourth aspect of the present invention, a text information extraction system is provided. A system includes a text detection device according to a second aspect of the present invention configured to detect a text region from an input image or an input video, and an extraction device configured to extract text from the detected text region. And a recognition device configured to recognize the extracted text and acquire text information.

  By utilizing these features, the method, apparatus and system according to the present invention can quickly and / or accurately indicate the location of text in an image, resulting in improved text detection performance. .

  Further features and advantages of the present invention will become apparent from the following description with reference to the drawings.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
FIG. 1 is a flowchart illustrating a prior art method for detecting a text region in an image. FIG. 2 is a block diagram illustrating an exemplary hardware configuration of a computer system that can implement an embodiment of the present invention. FIG. 3 is a flowchart illustrating a text detection method for detecting a text region in an image according to an embodiment of the present invention. FIG. 4 is a diagram illustrating an exemplary process for determining whether a CC is a TBR based on a relationship between the CC and another TBR according to an embodiment of the present invention. FIG. 5A is a flowchart illustrating the CC filtering step of FIG. 3 according to an embodiment of the present invention. FIG. 5B is a flowchart illustrating the CC filtering step of FIG. 3 according to another embodiment of the present invention. FIG. 5C is a diagram illustrating exemplary processing in the CC filtering step of FIG. 3 using a training classifier according to an embodiment of the present invention. FIG. 6 is an exemplary flowchart illustrating performing the CC grouping step of FIG. FIG. 7 is an exemplary flowchart illustrating performing the grouping steps of FIG. , , , , , 8A to 8F are diagrams illustrating exemplary images showing respective processing results using the text detection method according to the present invention. FIG. 9 is a block diagram illustrating a text detection device that detects a text region in an image according to an embodiment of the present invention. 10A is a block diagram illustrating the CC filtering unit of FIG. 9 according to an embodiment of the present invention. 10B is a block diagram illustrating the CC filtering unit of FIG. 9 according to another embodiment of the present invention. FIG. 11A is a block diagram illustrating the CC grouping unit of FIG. 9 according to an embodiment of the present invention. FIG. 11B is a block diagram illustrating the grouping unit of FIG. 11A according to one embodiment of the present invention. FIG. 12 is a flowchart illustrating a text information extraction method according to an embodiment of the present invention. FIG. 13 is a block diagram showing a text information extraction system according to an embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

  In addition, since the same code | symbol and alphabet in a figure show the same item in a figure, if an item is prescribed | regulated in one drawing, it is not required to be demonstrated with respect to subsequent drawing.

  Also, in the present invention, terms such as “first” and “second” are used only to distinguish elements or steps and are not intended to indicate time order, presence or importance.

  FIG. 2 is a block diagram showing a hardware configuration of a computer system 1000 that can implement the embodiment of the present invention.

  As shown in FIG. 2, the computer system includes a computer 1110. For example, the computer 1110 may be a digital camera or a smartphone. The computer 1110 includes a processing unit 1120, a system memory 1130, a non-removable non-volatile memory interface 1140, a removable non-volatile memory interface 1150, a user input interface 1160, a network interface 1170, and a video interface 1190. Output peripheral interface 1195, which are connected via a system bus 1121.

  The system memory 1130 includes a ROM (read only memory) 1131 and a RAM (random access memory) 1132. A BIOS (basic input / output system) 1133 resides in the ROM 1131. Operating system 1134, application program 1135, other program modules 1136, and some program data 1137 reside in RAM 1132.

  A non-removable non-volatile memory 1141 such as a hard disk is connected to a non-removable non-volatile memory interface 1140. Non-removable non-volatile memory 1141 can store, for example, operating system 1144, application programs 1145, other program modules 1146, and some program data 1147.

  One or more removable non-volatile memory drives, such as flash drive 1151 and CD-ROM drive 1155, are connected to removable non-volatile memory interface 1150. For example, a flash memory 1152 such as an SD card can be inserted into the flash drive 1151, and a CD (compact disk) 1156 can be inserted into the CD-ROM drive 1155. The image to be processed can be stored in a non-volatile memory.

  Input devices such as a microphone 1161 and a keyboard 1162 are connected to the user input interface 1160.

  The computer 1110 can be connected to the remote computer 1180 via the network interface 1170. For example, the network interface 1170 can be connected to the remote computer 1180 via the local area network 1171. Alternatively, the network interface 1170 can be connected to a modem (modulator-demodulator) 1172, and the modem 1172 is connected to the remote computer 1180 via the wide area network 1173.

  The remote computer 1180 can include a memory 1181 such as a hard disk that stores the remote application program 1185.

  The video interface 1190 is connected to a monitor 1191 that may be used to display one or more processing results according to embodiments of the present invention.

  The output peripheral interface 1195 is connected to the printer 1196 and the speaker 1197.

  The computer system shown in FIG. 2 is merely exemplary and is in no way intended to limit the invention, its application, or use.

  The computer system shown in FIG. 2 is a stand-alone computer or device processing system, possibly with one or more unnecessary components removed or with one or more additional components added. It may be realized for any of the embodiments.

  FIG. 3 is a flowchart illustrating a text detection method 300 for detecting a text region in an image according to an embodiment of the present invention. Further description will be made with reference to FIGS. 8A to 8F showing processing results using the text detection method according to the present invention.

  According to one embodiment, the image includes at least one connected component (CC). CC is a pixel cluster (pixel group) including similar colors or gray scale values. Pixels in one cluster can be connected in the vicinity of 4 or 8 in space. The CC can be generated from the image before the TBR detection step, for example by color clustering, adaptive binarization and morphological processing. In one embodiment, “Robide wide baseline from stable stable extreme regions,” J. Pat. Matas, O.M. Chum, M.M. Urban and T.W. Pajdla, Proc. Based on the MSER method described in of the British Machine Vision Conference, pages 384-396, CCs can be generated from grayscale images. In one embodiment, in order to detect both light text on a dark background and dark text on a light background, CC generation is performed on two channels, one for the original image and the other for the inverse image. It is applicable to. However, this is not always required.

  As an example, FIGS. 8A and 8B show a grayscale image and the generated CC in the image, respectively. In FIG. 8B, each black line box represents a CC. In other words, the black line box is a bounding box of CC (the smallest square including CC).

  In block 310, a text background region (TBR) detection step is performed to detect at least one TBR from the input image.

  According to one embodiment, the TBR detection step may be performed based on the CC included in the image. The TBR may be a CC that includes special characteristics. According to one embodiment, the TBR may be a surrounding area of text in an image having regular borders and a uniform color or gray scale. In a natural scene image, TBR usually exists as a signboard, logo, tag, scroll, poster, etc., but is not limited to these forms.

  According to one embodiment, the TBR may be selected from CCs included in the image. Each CC can be checked to determine if it is a TBR. Three aspects of characteristics may be considered in determining CC characteristics, statistical characteristics of member CCs in the CC, and relationships between CCs and other TBRs. These aspects can be used individually or in any combination for the detection of TBR. Here, the member CC in the current CC is a CC arranged within the boundary of the current CC, and has a high contrast with respect to the current CC. The member CC can be extracted from the channel on the other side of the current CC. For example, a dark CC member CC may be a bright CC in a dark CC region, and a bright CC member CC may be a dark CC in a bright CC region.

  FIG. 8C shows an image in which two TBRs are detected, indicated by a white line box.

[Characteristics of CC]
The CC can be checked based on its characteristics to determine if it is a TBR. CC features include, for example, CC color or gray scale uniformity, CC size, CC shape, CC boundary regularity, CC location in the image, CC average gray scale value and CC gray. At least one of the scale value distributions may be included.

  Note that the size of the TBR is usually relatively large. Thus, according to one embodiment, all CCs can be sorted by size, and the TBR can be selected from the top n largest CCs.

  Note that the TBR is usually arranged at a prominent position in the image, not in the blank area of the image. Therefore, according to another embodiment, CCs arranged in the blank area can be excluded as non-TBR areas. For example, the margin area can be defined as the outer loop area of an image having a specified width, such as a 1 / m image width, or a 1 / m image height.

  According to yet another embodiment, the regularity of the CC boundary can be considered in the determination of the TBR because the TBR typically has a regular boundary. The regularity of the boundary line is the density of CC (occupation ratio of CC in the bounding box), the boundary line ratio (ratio of the amount of boundary line pixels to the CC pixel), and the symmetry of the boundary line (density in four quadrants). Boundary similarity in four quadrants that can be evaluated by difference).

  According to yet another embodiment, some features are used to distinguish the TBR from the flash since the TBR should not be formed by the flash. It should be noted that flash typically has a high average gray scale value that includes a much higher gray scale value than the average in the middle. Thus, the average grayscale value and grayscale value distribution of CC can be used to distinguish TBR from flash.

[Statistics of member CC in CC]
A CC may be checked based on statistics of member CCs in the CC to determine if it is a TBR. Here, the member CC in the current CC is a CC arranged within the boundary of the current CC, and has a high contrast with respect to the current CC. The region of member CC is completely within the boundaries of the current CC. The member CC can be extracted from the channel on the other side of the current CC. For example, a dark CC member CC may be a bright CC in a dark CC region, and a bright CC member CC may be a dark CC in a bright CC region.

  The statistics of the member CC include, for example, the number of member CCs in the CC, the number of seed CCs of the member CC, the average text reliability of the member CC in the CC, and the ratio of the total area of the member CC and the area of the CC in the CC. At least one of them.

  For the sake of explanation, it is preferable that the number of member CCs in the CC is larger than the threshold.

  The number of seed CCs among the members CC is preferably larger than a threshold value. Here, the seed CC indicates a CC that is very likely to be a text component. For example, the seed CC may be a CC that has a text confidence higher than a predefined threshold. In order to select a seed CC, a set of features can be extracted from the CC to calculate the text reliability of the CC.

  As an example, FIG. 8D shows several seed CCs in a white box.

  Commonly used features for selecting the seed CC are: CC size, CC width / height ratio, CC density (ie occupancy of CC pixels in the bounding box), CC stroke width Statistical features and texture features extracted from the CC region. In one embodiment, features can be used as rules in adjusting text confidence. In another embodiment, the text classifier can be learned based on a training set that includes both text CC and non-text CC. The classifier uses CC features as input and outputs CC text confidence values.

[Relationship between CC and other TBRs]
A CC can be checked based on relationships with other TBRs to determine if it is a TBR. According to one embodiment, selecting a TBR from a CC based on the relationship between at least one CC and another TBR is not a member CC in any previously determined TBR, and the previously determined TBR In response to CCs that do not have the same member CC, may include determining the CC as a TBR. In other words, TBRs typically do not overlap with each other or do not include each other.

  As an example, FIG. 4 shows a process 400 for determining whether a CC is a TBR based on the relationship between the CC and another TBR.

  In block 410, it is determined whether the current CC is a previously determined member BR of the TBR. The current CC is identified as not being a TBR if it is a member CC of a previously determined TBR. If the current CC is not a previously determined TBR member CC, process 400 proceeds to block 420 to determine if the current CC has the same member CC as the previously determined TBR. The current CC is identified as a TBR if it does not have any member CC identical to the previously determined TBR. If the current CC has the same member CC as the previously determined TBR, process 400 proceeds to block 430 to determine which of the current CC or the previously determined TBR is more like a TBR. To do. If the current CC is more like a TBR, the previously determined TBR is removed from the set of TBRs (block 440) and the current CC is identified as a TBR. If the previously determined TBR is more like a TBR, the current CC is identified as not being a TBR.

  There may be various ways to determine which is more like TBR at block 430. For example, one or more of the above criteria may be used, such as boundary regularity, CC density, and member CC average text reliability.

  FIG. 3 will be referred to again. Since TBR has been detected from the image, process 300 proceeds to block 320. In block 320, a CC filtering step is performed to filter at least one CC to ensure at least one candidate text CC.

  In particular and preferably, after selecting a TBR from a CC, the remaining CCs are non-TBR CCs including candidate text CCs and non-text CCs. After this step, candidate text CCs are reserved, but non-text CCs are removed.

  According to one embodiment, TBR information is used to filter CCs. Both CCs within the TBR boundary and CCs not within any TBR boundary are preferably filtered based on the same rule. For example, a CC within a TBR boundary and a CC that is not within any TBR boundary are: CC size, CC shape, CC bounding box aspect ratio, CC to its bounding box area ratio, perimeter and CC Based on at least one of the area ratio and the texture feature of CC.

  Preferably, when filtering CC, the size of CC may be selected as being in a predetermined range based on experience values. Since the aspect ratio of the text CC is usually not too high, the aspect ratio of the bounding box of the CC can be selected as being lower than a predetermined threshold. The area ratio between the CC and its bounding box should not be too low and can be selected as higher than a predetermined threshold according to experience. The circumference with respect to the area ratio of CC can be shorter than a predetermined threshold. This is because the perimeter of the noise CC relative to the area ratio is usually relatively long, so that the noise CC is removed. Texture features such as wavelets, Gabor, LBP, etc. extracted from the CC can be used to calculate the text reliability and can therefore be incorporated when filtering the CC.

  Note that text is more likely to appear in the TBR than outside the TBR. Thus, according to another embodiment, TBR information can be used when filtering CCs to further improve the efficiency and accuracy of filtering.

  In particular, the filtering for CCs that are not within the boundaries of any TBR, for example in the CC filtering step, may be stricter than the filtering for CCs within the boundaries of the TBR. In another example, CCs that are not within the boundaries of any TBR may be filtered by more rules than CCs that are within the boundaries of the TBR.

  This is because it is likely to suffer noise as a result of having a relatively low text reliability with which the CC determined to be not within the boundaries of any TBR is calculated. Thus, fine filtering can be performed to remove non-text CCs.

  Preferably, CCs that are not within the boundaries of any TBR can be filtered further based on stroke width statistics and / or the ratio of CC border pixels to CC pixels. For example, the stroke width statistics may include a ratio of stroke width variance to average stroke width.

  FIG. 8D shows the result of filtering. The reserved text candidate CC is indicated by a bounding box drawn with a black line or a white line. Here, a white line bounding box indicates a seed CC, and a black line bounding box indicates a non-seed CC. Compared to FIG. 8B, a part of the CC, particularly the CC arranged outside the TBR, is removed.

  In order to illustrate an exemplary method for performing CC filtering, two embodiments are described below.

  FIG. 5A is a flowchart illustrating the CC filtering step of FIG. 3 according to an embodiment of the present invention.

  In FIG. 5A, the text reliability for each of the non-TBR CCs is calculated based on whether the CC is located within the boundaries of every TBR.

  At block 510, the text reliability of each non-TBR CC is calculated based on the TBR information. In calculation, CCIR is more important than CCOR.

  At block 520, it is determined whether the text reliability of the CC is higher than a predefined threshold T. If the text reliability is higher than the threshold, the CC is determined as a text candidate CC. If the text reliability is lower than the threshold, the CC is determined as a non-text CC.

A specific example of this embodiment is provided as follows. The text reliability of the current CC can be defined by Bayes' theorem.
P (A | B) = P (B | A) P (A) / P (B)
Where P (A) is the prior probability, P (B | A) is the conditional probability, and P (A | B) is the posterior probability.

  Considering a specific example of CC filtering, A is a random variable that indicates a particular CC label (text or non-text). P (A) represents the prior probability of text presence of the current CC. P (A) can be determined by TBR. CCIR can be given a higher P (A) than CCOR.

  P (B | A) is the conditional probability of text presence. P (B | A) describes what the text area looks like. Thus, the value is calculated based on the text features extracted from the text area itself. P (B) is the existence probability of the current CC. When CC is fixed, P (B) is a constant value.

  P (A | B) is the text reliability of the current CC. P (A | B) is affected by both the text characteristics of the CC itself and the TBR information associated with the CC. CCs having a P (A | B) value higher than a predefined threshold are reserved as text candidate CCs.

  In this example, P (A) / P (B) for CCIR can be set as 1, and P (A) / P (B) for CCOR can be set as a value in the range of (0, 1). .

  FIG. 5B is a flowchart illustrating CC filtering according to another embodiment of the present invention.

  In FIG. 5B, filtering is performed on non-TBR CCs in several stages, eg, two stages, based on TBR information. For example, the two stages include coarse filtering and fine filtering. All non-TBR CCs are subject to coarse filtering, but only CCs outside any TBR boundary may be subject to fine filtering. Simple features can be used during coarse filtering, and more complex features can be used during fine filtering. Therefore, filtering for CCs that are not within the boundaries of any TBR is more severe than filtering for CCs that are within the boundaries of the TBR. Thus, the amount of CC subjected to fine filtering is reduced and the efficiency of the method is improved.

  At block 530, the non-TBR CCs are separated into two groups: CCIRs, ie CCs within any TBR boundary, and CCORs, CCs not within any TBR boundary.

  At block 540, a first filtering step, such as coarse filtering, is performed for all non-TBR CCs. In particular, each non-TBR CC is determined as a candidate text CC or non-text CC.

  The first filtering step may be performed based on one or more first features of each of the non-TBR CCs to determine whether the CC is a candidate text CC. The first feature is extracted from the CC, the size of the CC, the shape of the CC, the aspect ratio of the bounding box of the CC, the density of the CC (the area ratio of the CC and its bounding box), the perimeter to the area ratio of the CC, and the CC May be relatively simple features including, but not limited to. As an example, texture features may include, but are not limited to, local binary patterns, edge direction histograms, and gradient histograms.

  The first feature can be used as a cascade rule or can be combined as a feature vector that is input to a training classifier. Cascade rule thresholds or classifiers can be learned from both text samples and non-text samples. When cascade rules are used in filtering, each input CC can be checked by a predefined rule, and CCs that do not satisfy at least one of the rules are removed.

  In block 550, a second filtering step, such as fine filtering, is performed by the candidate text CCOR, ie, the first filtering step, to further determine whether the candidate text CCOR is a candidate text CC. It is executed for each CCOR determined as a text CC. Even if the CCOR is determined as a candidate text CC in block 540, the second filtering may further remove non-text CCs because it is susceptible to noise.

  In the second filtering step, more stringent conditions may be employed for the features used in block 540 and / or some other features may be used for filtering. Other features may include stroke width statistics (eg, ratio of stroke width variance to average stroke width) and / or ratio of borderline pixel number to CC pixel number).

  Similar to the first filtering step, the features for the second filtering can be used as cascade rules or can be combined as feature vectors that are input to the training classifier. Cascade rule thresholds or classifiers can be learned from both text samples and non-text samples. When cascade rules are used in filtering, each input CC can be checked by a predefined rule, and CCs that do not satisfy at least one of the rules are removed.

  FIG. 5C is a flowchart illustrating CC filtering using a training classifier. The method of FIG. 5C is applicable to both the first filtering step and the second filtering step. In FIG. 5C, training samples including text samples and non-text samples are used to train the classifier. CCs are classified by a training classifier to obtain text candidate CCs. The features extracted for training and classification are the features described above in connection with the first filtering and the second filtering. In order to avoid unnecessarily obscuring the subject matter of the present invention, no further details regarding the flowchart will be described.

  FIG. 3 will be referred to again. After obtaining the candidate text CC, the process 300 proceeds to block 330. In block 330, the CC grouping step groups at least one candidate text CC based on the TBR detected in the TBR detection step to form at least one CC group, and based on the at least one CC group Executed to generate at least one text region.

  An exemplary flowchart for performing the CC grouping step 330 is shown in FIG. As shown in FIG. 6, the CC grouping step 330 may include steps 610-630.

  The input to the CC grouping step is a candidate text CC.

  In step 610, candidate text CCs are assigned to respective text background regions. The text CC that is a candidate that cannot be assigned to the TBR is assigned to the outer area.

  Step 610 is similar to the step as shown in block 530 of FIG. 5B. Therefore, the description about step 610 is omitted. If no TBR is detected in step 310, all CCs are in the outer region. After assigning the candidate text CC, the process proceeds to step 620.

  In step 620, each TBR and outer region CCs are grouped together to form a CC group.

  In this step, the CCs in one region are grouped based on spatial relationships and appearance similarities. An exemplary flowchart for performing the grouping step 620 of FIG. 6 is shown in FIG.

  As shown in FIG. 7, the grouping step 620 may include steps 6201-6203.

  In step 6201, each TBR and outer region CC is divided into a dark region set of bright CCs and a bright region set of dark CCs.

  In step 6202, CC groups are created in the bright CC and dark CC sets, respectively.

  According to a preferred embodiment, CC groups can be generated by CC clustering. CC clustering is the alignment of CC centers according to a certain direction, CC size similarity, CC shape similarity, CC color or grayscale similarity, CC stroke width similarity and between CCs. Use one or more of the distance constraints.

  According to another embodiment, the CC group is first generated by a Hough transform. As a result of the Hough transform, CCs centered on one row are grouped together. Next, CCs included in the generated CC group are filtered using the above constraints.

  In step 6203, the bright CC and dark CC are grouped in step 6202, respectively, and then the bright CC set and dark CC set are combined based on spatial relationships and / or appearance similarities.

  Features commonly used to combine CC groups in step 6203 are, for example, the bounding box overlap rate of two specific groups, the size similarity (the difference in size is the height of the two groups Lower than the maximum value) and row direction similarity (direction difference is preferably less than 30 degrees). One or any combination of the above features may actually be used.

  After the completion of step 6203, each TBR and outer area CC group are formed. Proceed to step 630 in FIG.

  In step 630, CC groups from various regions between each TBR and the outer region are combined to produce a text region.

  In this step, CC groups from different regions are classified into row direction consistency, CC average size similarity, CC average stroke width similarity and CC average color or grayscale similarity. Based on at least one of

  The rules for combining CC groups from different regions are preferably stricter than the rules for grouping CC groups from one region. Since CC groups from various regions do not overlap each other, the overlapping rate of bounding boxes may not be used.

  CC grouping step 330 is completed when step 630 is completed.

  FIG. 8E shows the result of the CC grouping step 330. It can be seen from FIG. 8E that candidate text CCs are grouped into text lines / words to which CCs belonging to the same string are connected using white lines.

  The text detection method 300 ends when the CC grouping step 330 is completed.

  FIG. 8F shows the result of the text detection method 300. It can be seen that the text region based on the text line / word bounding box has been detected from the input image. The last detected text area is indicated by a white line box.

  Reference is now made to FIG. 9 showing a block diagram of a text detection device 900 that detects text regions in an image according to an embodiment of the present invention. The apparatus 900 can be used to implement the method described with reference to FIGS. For brevity, some details similar to those described with reference to FIGS. 3-7 are omitted here. However, it will be understood that these details may also be applicable to apparatus 900.

  According to one embodiment as shown in FIG. 9, the text detection device 900 comprises a text background region (TBR) detection unit 910, a CC filtering unit 920, and a CC grouping unit 930.

  Text background region (TBR) detection unit 910 may be configured to detect TBR from an image.

  CC filtering unit 920 may be configured to filter at least one CC to ensure at least one candidate text CC.

  The CC grouping unit 930 groups at least one candidate text CC based on the TBR detected in the TBR detection unit to form at least one CC group, and at least one based on the at least one CC group. Can be configured to generate one text region.

  As used herein, a TBR may be defined as the surrounding area of text in an image and has a regular border and a uniform color or gray scale.

  According to one embodiment, the TBR detection unit 910 is a CC located within the boundary of at least one CC, at least one CC boundary, and in at least one CC having a high contrast to the at least one CC. May be configured to select a TBR from at least one CC based on at least one of the statistics of the member CCs and the relationship between the at least one CC and another TBR.

  For example, at least one CC feature includes CC color or gray scale uniformity, CC size, CC shape, CC border regularity, CC location in the image, CC average gray scale value, and It may include at least one of a CC grayscale value distribution.

  For example, the statistics of member CC may include the number of members CC in the CC, the number of seed CCs of members CC that have a text confidence higher than the first predefined threshold, the average text confidence of members CC in the CC, It may include at least one of the ratio of the total area of the members CC in CC and the area of CC.

  For example, selecting a TBR from at least one CC based on the relationship between at least one CC and another TBR is not a member CC in any previously determined TBR, and the previously determined TBR And determining a CC as a TBR in response to a CC that does not have the same member CC. This can be realized by executing the flowchart as shown in FIG. 4 by the TBR detection unit 910.

  According to one embodiment, in CC filtering unit 920, filtering for CCs that are not within the boundaries of any TBR may be stricter than filtering for CCs that are within the boundaries of any TBR.

  According to one embodiment, the CC filtering unit 920 includes the following conditions: CC size, CC shape, CC bounding box aspect ratio, CC to its bounding box area ratio, perimeter and CC area. Based on at least one of the ratio and the texture feature of the CC, it is configured to filter CCs within any TBR boundary and CCs not within any TBR boundary. The CC filtering unit 920 is further configured to filter CCs that are not in any TBR based on at least one of the following conditions: stroke width statistics and CC border pixel number to CC pixel number. The

  10A is a block diagram illustrating the CC filtering unit of FIG. 9 according to an embodiment of the present invention.

  As shown in FIG. 10A, according to one embodiment, the CC filtering unit 920 is a calculation unit 1010 configured to calculate the text reliability of each of at least one CC other than TBR, A calculation unit 1010 in which CCs in every TBR boundary are more important than other CCs, and a determination unit 1020 configured to determine CCs having text reliability higher than a predefined threshold as text candidate CCs. Prepare.

  10B is a block diagram illustrating the CC filtering unit of FIG. 9 according to another embodiment of the present invention.

  As shown in FIG. 10B, according to another embodiment, the CC filtering unit 920 is responsive to CCs being placed within the boundaries of every TBR for every at least one CC other than a TBR. In order to determine whether the CC is a text candidate CC and an identification unit 1102 configured to identify the CC as a first CC or a CC as a second CC A first filtering unit 1104 configured to perform a first filtering step for each of the two CCs, and a first filtering step to further determine whether the CC is a text candidate CC. Configured to perform a second filtering step for each second CC determined as a text candidate CC. And a second filtering unit 1106.

  According to one embodiment, the first filtering unit 1104 performs a first filtering step based on one or more first characteristics of the CC so as to determine whether the CC is a text candidate CC. Can be further configured.

  According to one embodiment, the second filtering unit 1106 performs a second filtering step based on one or more second characteristics of the CC to further determine whether the CC is a text candidate CC. Can be further configured.

  FIG. 11A is a block diagram illustrating the CC grouping unit 930 of FIG. 9 according to an embodiment of the present invention.

  According to an embodiment, the CC grouping unit 930 may further comprise an allocation unit 9301, a grouping unit 9302, and a first combination unit 9303.

  The assignment unit 9301 may be configured to assign candidate text CCs to respective text background regions and assign candidate text CCs that cannot be assigned to TBRs to outer regions.

  The grouping unit 9302 may be configured to group CCs in each TBR and outer region to form a CC group.

  The first combination unit 9303 may be configured to combine the CC groups from various regions between each TBR and the outer region to generate the at least one text region.

  According to one embodiment, the first combination unit 9303 includes the following conditions: row direction consistency, CC group group bounding box overlap rate, CC average size similarity, CC average stroke width similarity. Based on at least one of sex and the average color or grayscale similarity of CCs, it can be configured to combine CC groups from different regions.

  FIG. 11B is a block diagram illustrating the grouping unit 9302 of FIG. 11A according to one embodiment of the invention.

  According to an embodiment, the grouping unit 9302 may further comprise a split unit 9302-1, a generation unit 9302-2, and a second combination unit 9302-3.

  The division unit 9302-1 may be configured to divide each TBR and outer area CC into a dark CC set of bright CCs and a dark CC set of bright areas.

  Generation unit 9302-2 may be configured to generate CC groups within a set of bright CCs and a set of dark CCs, respectively.

  Second combination unit 9302-3 may be configured to combine a set of bright CCs and a set of dark CCs based on at least one of spatial relationships and appearance similarities.

  According to an embodiment, in the generation unit 9302-2, the CC group may be generated by CC clustering. CC clustering consists of the following constraints: CC center alignment according to a certain direction, CC size similarity, CC shape similarity, CC color or grayscale similarity, CC stroke width At least one of similarity and distance between CCs may be used.

  According to one embodiment, the generating unit 9302-2 generates a CC group by Hough transform, and the following constraints: CC center alignment according to a certain direction, CC size similarity, CC shape Filtering CCs included in a CC group generated using at least one of: similarity, CC color or grayscale similarity, CC stroke width similarity, and distance between CCs Can be further configured.

  According to one embodiment, the rules for combining in the first combination unit 9303 may be stricter than the rules for grouping in the grouping unit 9302.

  The text detection method and apparatus according to the present invention have various adaptation examples. For example, it can be used in automatically extracting text information from images or video captured by a camera.

  FIG. 12 shows a text information extraction method according to an embodiment of the present invention.

  As shown in FIG. 12, in block 1210, a text region from an input image or input video is detected using the text detection method according to the text detection method described with reference to FIGS.

  At block 1220, text may be extracted from the detected text region. Optionally, as shown in block 1240, text in the input video can be tracked in detecting text regions from the input video.

  At block 1230, text recognition may be performed on the extracted text to obtain text information.

  Reference is now made to FIG. 13 showing a block diagram of a text information extraction system 1300 according to an embodiment of the present invention. System 1300 may be used to implement the method described with reference to FIG.

  As illustrated in FIG. 13, the system 1300 includes a text detection device 1310, an extraction device 1320, and a recognition device 1330.

  The text detection device 1310 is configured to detect a text region from an input image or video and may be the same as the device 910 described in connection with FIG.

  The extraction device 1320 can be configured to extract text from the detected text region.

  The recognition device 1330 may be configured to recognize the extracted text and acquire text information.

  Optionally, system 1300 can further comprise a tracking device 1340. The tracking device 1340 may be configured to track text in the input video when the text detection device 1310 is configured to detect a text region from the input video.

  It will be appreciated that the units and devices described above in connection with FIGS. 9-11 and 13 are exemplary modules and / or preferred modules that implement various steps. A module may be a hardware unit (eg, a processor or an application specific integrated circuit) and / or a software module (eg, a computer program). The modules that implement the various steps are not fully described above. However, if there is a step of executing a specific process, there may be a corresponding functional module or functional unit (implemented by hardware and / or software) that implements the same process. As long as the technical solutions constituted by all combinations of the steps described above and below and the units corresponding to these steps are complete and applicable, the technical solutions by them are included in the disclosure content of the present invention. .

  In addition, the above-described apparatus and system configured by various units can be incorporated into a hardware device such as a computer as a functional module. Of course, the computer has other hardware or software components in addition to these functional modules.

  The method, apparatus and system of the present invention can be implemented in many ways. For example, the method and apparatus of the present invention can be performed by software, hardware, firmware, or any combination thereof. The above order of method steps is intended to be exemplary only, and unless otherwise indicated, the method steps of the present invention are not particularly limited to the order described above. In addition, in some embodiments, the present invention may also be implemented as a program recorded on a recording medium that includes machine-readable instructions for implementing the method according to the present invention. Therefore, the present invention also includes a recording medium that stores a program for realizing the method according to the present invention.

  While several specific embodiments of the present invention have been demonstrated in detail using examples, those skilled in the art will appreciate that the above examples are intended to be illustrative only, rather than limiting the scope of the invention. It should be. It should be understood by those skilled in the art that the embodiments described above can be modified without departing from the scope and spirit of the invention. The scope of the present invention is defined by the appended claims.

Claims (23)

A text detection method for detecting a text region in an image including at least one connected component (CC) comprising:
A TBR detection step (310) for detecting a text background region (TBR) from the image;
CC filtering step (320) for filtering the at least one CC to ensure at least one candidate text CC;
Grouping the at least one candidate text CC based on the TBR detected in the TBR detection step to form at least one CC group, and at least one text region based on the at least one CC group CC grouping step (330) for generating
A method comprising the steps of:   The method of claim 1, wherein the TBR is a surrounding area of text in the image and has a regular border and a uniform color or gray scale. The TBR detection step (310) includes:
The at least one CC feature;
Statistics of member CCs in the at least one CC that are located within the boundary of the at least one CC and have a high contrast to the at least one CC, and the at least one CC and other TBRs The relationship between
3. The method of claim 1 or 2, comprising selecting the TBR from the at least one CC based on at least one of the following. The characteristics of the at least one CC are:
CC color or gray scale uniformity,
The size of the CC,
The shape of the CC,
Regularity of the boundary line of the CC,
The position of the CC in the image,
The method of claim 3, comprising at least one of an average grayscale value of the CC and a grayscale value distribution of the CC. The statistics of the member CC are
The number of the member CCs in the CC;
The number of seed CCs of the member CC having a text confidence higher than a first predefined threshold;
The average text reliability of the member CC in the CC, and a ratio of the total area of the member CC and the area of the CC in the CC. the method of. Selecting the TBR from the at least one CC based on the relationship between the at least one CC and another TBR;
Determining a CC as a TBR in response to a CC that is not a member CC in any previously determined TBR and does not have the same member CC as the previously determined TBR. The method of claim 3.   The method according to claim 1 or 2, wherein in the CC filtering step (320), filtering for CCs not within the boundaries of any TBR is more strict than filtering for CCs within any TBR boundary. In the CC filtering step (320), CCs within any TBR boundary and CCs not within any TBR boundary are:
The size of the CC,
The shape of the CC,
The aspect ratio of the bounding box of the CC,
The area ratio of the CC and its bounding box,
The ratio of the perimeter to the area of the CC, and the texture features of the CC,
The method of claim 1, wherein the filtering is based on at least one of the following: The CC not in any TBR is
Stroke width statistics, and the ratio between the number of border pixels and CC pixels,
The method of claim 8, further filtered based on at least one of the following: The CC grouping step (330) includes:
a) assigning the candidate text CC to each text background region and assigning the candidate text CC that cannot be assigned to a TBR to an outer region (610);
b) grouping each TBR and CC of the outer region to form a CC group (620);
and (c) combining the CC groups from various regions between each TBR and the outer region to generate the at least one text region (630). Method. Said step b)
b1) dividing (6201) each TBR and CC of the outer region into a set of dark CCs that are bright and a set of dark CCs that are bright;
b2) generating a CC group in the set of bright CCs and the set of dark CCs, respectively (6202);
The method of claim 10, further comprising: b3) combining (6203) the set of bright CCs and the set of dark CCs based on at least one of spatial relationship and appearance similarity. The method described. In step c)
Row-wise consistency,
Similarity of the average size of the CC,
The average stroke width similarity of the CC, and the average color or grayscale similarity of the CC,
11. The method of claim 10, wherein CC groups from different regions are combined based on at least one of the following. In step b2), the CC group is generated by CC clustering,
The CC clustering has the following constraints:
CC center alignment according to a certain direction,
CC size similarity,
CC shape similarity,
CC color or grayscale similarity,
The method according to claim 11, wherein at least one of a stroke width similarity of CCs and a distance between CCs is used. The step b2)
Generating a CC group by Hough transform;
Filtering CCs included in the generated CC group using at least one of the following constraints:
CC center alignment according to a certain direction,
CC size similarity,
CC shape similarity,
CC color or grayscale similarity,
The similarity of stroke width of CC and the distance between CC
The method according to claim 11.   11. The method of claim 10, wherein the rules for combining in step c) are stricter than the rules for grouping in step b). The CC filtering step includes:
Calculating a text reliability of each of the at least one CC other than the TBR;
Here, in the calculation, CCs within the boundary of every TBR are more important than other CCs.
And determining a CC having a text confidence higher than a second predefined threshold as the text candidate CC. The CC filtering step includes:
For each at least one CC other than the TBR, in response to the CC being located within the boundary of every TBR, identify the CC as a first CC, or identify the CC as a second CC Identifying as CC;
Performing a first filtering step on each of the first CC and the second CC to determine whether the CC is the text candidate CC;
In order to further determine whether the second CC is the text candidate CC, a second filtering step for each of the second CCs determined as the text candidate CC by the first filtering step. The method of claim 1 comprising the steps of: Performing the first filtering step for each of the first CC and the second CC comprises:
18. The step of performing the first filtering step based on one or more first characteristics of the CC to determine whether the CC is the text candidate CC. The method described in 1. Performing the second filtering step on each of the second CCs determined as the text candidate CC by the first filtering step,
Performing the second filtering step based on one or more second features of the second CC to further determine whether the second CC is the text candidate CC. The method of claim 17, wherein: The texture features are:
Local binary pattern,
The method of claim 8, comprising at least one of an edge direction histogram and a gradient histogram.   19. The method of claim 18, wherein in the first filtering step, the first feature is used as a cascade rule or combined as a feature vector input to a training classifier.   20. The method of claim 19, wherein in the second filtering step, the second feature is used as a cascade rule or combined as a feature vector that is input to a training classifier. A text detection device for detecting a text region in an image including at least one connected component (CC) comprising:
The TBR detection unit (910) configured to detect a text background region (TBR) from the image;
A CC filtering unit (920) configured to filter the at least one CC to ensure at least one candidate text CC;
Grouping the at least one candidate text CC based on the TBR detected in the TBR detection unit to form at least one CC group, and at least one text region based on the at least one CC group CC grouping unit (930) configured to generate
A device comprising: