JP2003515229A - Symbol classification with shape features given to neural networks - Google Patents

Abstract

(57) Abstract An image processing apparatus and method for classifying symbols, such as text, in a video stream uses a back-propagation neural network in which a feature space is obtained from size-, translation-, and rotation-invariant shape-dependent features. Various exemplary feature spaces discuss, for example, histograms of angles resulting from Delaunay triangulations of refinement symbols that are regular and have constant moments and thresholds. Such a feature space provides a good match for BPNN as a classifier due to the poor resolution of characters in the video stream.

Description

Detailed Description of the Invention

[0001]

TECHNICAL FIELD OF THE INVENTION

The present invention is described in US patent application Ser. No. 09 / 370,931 entitled "System and Method for Analyzing Video Content Using Text Detected in Video Frames", filed August 9, 1999. Regarding The above applications are commonly assigned to the assignee of the present invention and are hereby incorporated by reference in their entirety as if fully set forth herein. The present invention is also disclosed in US Preliminary Patent Application No. 60 / 117,658 entitled "Method and Apparatus for Text Detection and Localization in Video", filed January 28, 1999. It also concerns. The above-referenced preliminary patent application is commonly assigned to the assignee of the present invention, and the disclosure content of the preliminary patent application is, for all purposes, fully described herein by reference. As such is hereby incorporated by reference. The invention also relates to what is disclosed in the application entitled "Classification of symbols with shape features supplied to neural networks" filed at the same time as the present application. This application is commonly assigned to the assignee of the present invention, and the disclosure content of the related preliminary patent application is, for all purposes, as if fully set forth herein by reference. Incorporated herein.

[0002]

BACKGROUND OF THE INVENTION

This invention relates to systems for recognizing patterns in digitized images, and more particularly to systems for separating symbols such as text characters in video data streams.

Real-time broadcasting, analog tape and digital video are important for a host of educational, entertainment and multimedia applications. Given the size of video collections, which can reach millions of hours, techniques for interpreting video data are needed to enable such material to be used and accessed more effectively. Various such advanced uses have been proposed. For example, the use of text and voice recognition can lead to the creation of a digest of the original video and the automatic generation of keys to index the video content. Another series of applications was broadcast (
Relies on fast real-time classification of text and / or other symbols in a video data stream (or otherwise multicast). For example, text recognition can be used for any suitable purpose, such as indexing video content.

Various text recognition techniques have been used to recognize digitized patterns. The most common example is optical character recognition (OCR) of documents. The general model of these techniques is that an input vector is extracted from the image and that the input vector characterizes the raw pattern. The vector is mapped to a fixed number or series of symbol classifications to "recognize" the image. For example, the pixel values of the bitmap image can serve as the input image, and the corresponding set of classifications can be, for example, the alphabet, for example the English alphabet. No particular technique for pattern recognition has achieved universal superiority. Each cognitive problem has its own set of application difficulties. That is, the size of the classification set, the size of the input vector, the required speed and accuracy, and other issues. Further, reliability is also an area where improvement is eagerly desired in almost all application fields.

As a result of the shortcomings mentioned above, pattern recognition is an area of continuous active development, where different applications have received varying degrees of attention based on perceived advantages such as usefulness and practicality. There is. Perhaps the most mature of these techniques is the application of pattern recognition to text characters, or optical character recognition (OCR). This technology
It has evolved due to the desirability and practicality of converting printed matter into computer-readable characters. From a practical point of view, printed documents provide a relatively clear and consistent source of data. Such documents are generally characterized by high contrast patterns against a uniform background and can be stored at high resolution. For example, a printed document can be scanned at any resolution to form a binary image of printed characters. Also, for such pattern recognition applications, computer-based conversion of documents into text avoids the task of keyboard transcription, realizes the economics of data storage and allows documents to be retrieved. There is a clear demand for doing so.

[0006] Some application areas have received little attention due to the attendant difficulties of performing symbolic or character classification. For example, pattern recognition in a video stream is a difficult area due to at least the following factors. That is, characters in a video stream tend to be presented with poor resolution and low contrast against a spatially non-uniform (sometimes time varying) background. Therefore, character recognition in a video stream is difficult and a reliable method is unknown. In addition, for some applications, at least a fast recognition speed is highly desirable, as disclosed in the above-mentioned related applications.

Systems and methods for indexing and classifying videos are described in numerous publications, including: That is, "ConIVAS: Content-Based Image and Video Access Systems" by M. Abdel-Mottaleb et al., Pp. 427-428, 1996 ACM Multimedia, Boston; ACM, Seattle, 1994.
Multimedia bulletin, pages 313-324, SF. Chang et al., "Video Q:
Automated Content-Based Video Retrieval System Using Visual Cues "; 1995, ACM Comm. Vol. 38, No. 4, M. Christ, 57-58.
"Infomedia Digital Video Library" by el et al .; IEEE Bulletin on Knowledge and Data Engineering, November 1998, "Video Content Management on Consumer Devices" by N. Dimitrova et al., August 1998. Brisbane's International Conference on Pattern Recognition, 916-918, by U. Gargi et al.
Indexing Text Events in Digital Video Databases "; 1996
August 2015, IEEE Electronics Bulletin, Volume 42, Issue 3, MK Mandal et al., "Indexing Images Using Moments and Wavelets," and 1996 Journal on Visual Communication and. Image Representation, Volume 7, Issue 4, 345-3
"Automatic Summary of Digital Movies" by S. Pfeiffer et al., Page 53.

Extraction of characters by a method using local thresholding and detection of image areas containing characters by assessing differences in gray levels between adjacent areas is described in February 1994 for pattern analysis and machine intelligence. IEEE Bulletin on Volume 16, Volumes 214-2
See "Recognition of Characters in Scene Images" by Ohya et al., Page 24. Ohya
Others further disclose integrating detection regions with close proximity and similar gray levels to generate character pattern candidates.

Using the spatial context and high-contrast properties of video text to detect text, merging regions where horizontal and vertical edges are in close proximity to each other has been described in 1995 Language and A. Hauptmann et al., "Text, Audio and Vision on Video Segmentation: Infomedia Project" at the AAAI Fall 1995 Symposium on Computational Models for Visual Integration.
It is described in. R. Lienhart and F. Suber discuss a non-linear color system that reduces the number of colors in video images in "Automatic Text Recognition for Video Indexing" at the January 1996 SPIE Conference on Image and Video Processing. ing. The document describes a split / consolidate process to generate uniform segments with similar colors. Lienhart and Suber detect characters in a uniform area, including foreground characters, black and white or strict characters, characters of limited size and characters with high contrast compared to the surrounding area, It uses various heuristics.

The use of multi-valued image decomposition methods to locate text and separate the image into multiple real foreground and background images is described in the 1998 IEEE Pattern Recognition Bulletin, Volume 31, pp. 2055-2076, AK. It is described in "Automatic Text Search in Image and Video Frames" by Jain and B. Yu. JC. Shim et al. Found a uniform area in the 1998 International Conference on Pattern Recognition, Bulletin, 618-620, "Automatic Text Extraction from Video for Content-Based Annotation and Search." It describes the use of a generalized region labeling algorithm for segmenting and extracting text. The identified foreground images are clustered to determine the color and position of the text.

Another useful algorithm for image segmentation is KV Mardia et al., "The Spatial Threshold for Image Segmentation," 1988, IEEE Bulletin on Pattern Recognition and Machine Intelligence, Vol. 10, 919-927. Processing method ”, A. Per
ez et al., 1987, IEEE Bulletin on Pattern Recognition and Machine Intelligence, Vol. 9, pages 742-751, entitled "Iterative Thresholding Methods for Image Segmentation".

Various methods are known for locating text in digitized bitmaps. A technique is known in which character data is binarized to form an image that can be characterized as black-on-white, and character recognition is performed on a bitmap image. The text and other patterns in the video stream are predictable, easy to classify, large and clear, so that there is in principle insufficient information to classify without the aid of auxiliary contextual data. Rough, fast-passing and unpredictable slopes and positions that do not include. There are also ongoing studies to increase recognition speed and accuracy. Therefore,
In the state of the art, there is room for improvement, especially when applications such as video stream data place an excessive load on the state of the art.

[0013]

DISCLOSURE OF THE INVENTION

Briefly, image processing apparatus and methods for classifying symbols, such as text, in video streams use Back Propagation Neural Networks (BPNNs), where the feature space is derived from size-, translation- and rotation-invariant shape-dependent features. A feature space is usually the space occupied by the symbols to be recognized. The shape dependent feature may be a straight line portion of the symbol. When the shape-dependent features in this feature space are used, a histogram of angles used as a feature vector for the related symbol is formed. This feature vector is a BPNN that produces various candidate symbols.
Given to. BPNN is used in the present invention, while BPNN and its training employ known techniques, for example, as described in US Pat. No. 4,912,654. Various exemplary feature spaces discuss regular angle histograms of angles resulting from Delaunay triangulations of refinement symbols with regular moments and thresholds, for example. Such a feature space provides a good match for BPNN as a classifier due to the poor resolution of characters in the video stream. The shape-dependent feature space can be implemented by precise separation of character ranges using the techniques described above in this application.

The ability to detect and classify text that appears in a video stream has many applications. For example, video scenes and portions thereof can be characterized and indexed based on classifications derived from such text. This can lead to indexing, advanced search capabilities, annotation features, etc. In addition, the recognition of text in the video stream enables the presentation of context sensitive features, such as callable links to websites generated in response to the appearance of web addresses in the broadcast video stream. can do.

Text in video poses a very different problem than that of document OCR, a well-developed but still mature technology. Text in documents tends to be solid and of high quality. In video, scaled scene images may include noise and uncontrolled illumination. The characters that appear in the video can be of varying color, size, font, orientation, thickness and background, can be complex and time varying, and so on. Also, many applications for video symbol recognition require high speed.

The technique adopted by the present invention to classify video text uses an accurate and fast technique for symbol separation. The symbol bitmap is then used to generate a shape dependent feature vector, which is fed to the BPNN. While the feature vector gives a great emphasis to the overall image shape, it is relatively insensitive to the variability problem mentioned above. In the technique of separating character regions, connected component structures are defined based on the detected edges. Since edge detection results in far fewer pixels overall than binarizing the entire field occupied by the symbol, the process of generating connection components can be significantly faster. The choice of feature space also improves recognition speed. According to the simulated BPNN, the size of the input vector can significantly affect the throughput. It is very important that the components used are selective from the selected feature space. Of course, it is also possible to form a heterogeneous feature space by combining a mixture of different features such as moments and line segment features. Also, the economics of computation can be realized if the selected features share the computation steps.

The invention will now be described in connection with some preferred embodiments with reference to the accompanying drawings for a more complete understanding of the invention. Regarding the figures, the details shown are:
For purposes of illustration only, and for purposes of describing the preferred embodiment of the invention,
It is presented in order to provide what is believed to be the most effective and easily understandable description of the principles and ideas of the present invention. In this regard, the structural details of this invention are not intended to be shown in any more detail than is necessary for a basic understanding of the invention, and the description taken in conjunction with the drawings will explain to those skilled in the art the invention. It is clear how some forms of can be implemented in practice.

[0018]

DETAILED DESCRIPTION OF THE INVENTION

Referring to FIG. 1, an image text analysis system 100 includes a video processing device 110.
, Video source 180 and possibly monitor 185 to receive the video input and to generate and store textual information embedded in the video. The video processing device 110 inputs a video image, analyzes a frame, separates text and character regions, and classifies text and / or character regions according to the procedure detailed below. The video is provided by video source 180. Video source 18
0 is a VCR with an analog-to-digital converter (ADC), a disc with digitized video, a cable box with an ADC, a DVD or a CD-R
It can be any source of video data including OM drives, Digital Video Home Systems (DVHS), Digital Video Recorders (DVR), Hard Disk Drives (HDD), etc. Video source 180 may be capable of providing several short clips, or multiple clips containing long lengths of digitized video images. The video source 180 MPEG-converts the video data.
2 and MJPEG in any analog or digital format.

The video processing device 110 includes an image processor 120, a RAM 130, and a storage unit 14.
0, user I / O card 150, video card 160, I / O buffer 170 and processor bus 175. The processor bus 175 transfers data between various components of the video processing device 110. The RAM 130 further includes an image text workspace 132 and a text analysis controller 134. The image processor 120 provides overall control of the video processing device 110 and includes parsing the text in the video frame based on system selected and user selected attributes to perform the necessary image processing for the image text analysis system 100. To do. This includes performing edit processing, processing digitized video images for display on monitor 185 and / or storage in storage 140, and data between various components of image text analysis system 100. Including the transmission of. The requirements and capabilities for image processor 120 are well known in the art and need not be described in detail except as required by the present invention.

Image processor 120 may be implemented as a suitably programmed computer circuit. A series of instructions loaded into the program memory causes this computer circuit to perform the operations described below with reference to Figures 2-11B. The series of instructions may be loaded into the program memory by reading a carrier containing the series of instructions, such as a disc. This reading may be done via a communication network such as the Internet. In this case, the service provider makes these sets of instructions available to interested parties.

RAM 130 provides random access memory for temporary storage of data generated by video processing device 110, which is otherwise not provided by the components within the system. RAM 130 includes memory 132 for the image text workspace and text parsing controller 134, as well as other memory required by image processor 120 and associated devices. Image text workspace 132 represents the portion of RAM 130 in which the video image associated with a particular video clip is temporarily stored during the text parsing process. The image text workspace 132 allows a copy of a frame to be modified without affecting the original data so that the original data can be recovered later.

In one embodiment of the invention, the text parsing controller 134 is RAM 1
10 represents a storage-only portion of 30 that is executed by the image processor 120 or an application program that analyzes a video image based on user-defined text attributes. The text parsing controller 134 may perform well-known editing techniques such as morphing or boundary detection between scenes, and novel techniques for video text recognition relevant to the present invention. The text parsing controller 134 may be implemented as a program on a CD-ROM, computer diskette or other storage medium that can be loaded into a removable disk port elsewhere in the storage 140 or video source 180, such as a video source 180. You can also

Storage 140 is a removable disk (magnetic or optical) for permanent storage of programs and other data, including required video and audio data.
Have one or more disk systems including. Depending on the system requirements, the storage 140 may include a video source (or sources) and the rest of the system,
And can be configured to interface with one or more bidirectional buses for transmission of video and audio data from such sources and the rest of the system. The storage unit 140 can transmit data at a video speed if necessary. Storage 140 is also sized to provide sufficient storage for a few minutes of video for editing purposes, including text attribute analysis. Depending on the particular application and the capabilities of the image processor 120, the storage 140 may be configured to have the capability of storing multiple video clips.

The user I / O card 150 includes various user devices (or a plurality of devices) not shown.
Can interface with the rest of the image text analysis system 100. The user I / O card 150 converts the data input from the user device into the format of the interface bus 175 and transmits it to the image processor 120 or to the RAM 130 for later access by the image processor 120. The user I / O card 150 also transmits data to a user output device such as a printer (not shown). Video card 160 provides an interface between monitor 185 and video processing device 110 via data bus 175.

The I / O buffer 170 provides an interface between the video source 180 and the rest of the image text parsing system 100 via the bus 75. As mentioned above, the video source 180 has at least one bidirectional bus to interface with the I / O buffer 170. The I / O buffer 170 transfers data to / from the video source 180 at the required video image transfer rate. In the video processing device 110, the I / O buffer 170 stores the data input from the video source 180 to the storage unit 140, the image processor 120, or the RAM as necessary.
To 130. Simultaneous transmission of video data to the image processor 120 provides a means of displaying the video data as it is input.

Referring to FIGS. 2, 3A and 3B, a text extraction and recognition process 100 (as outlined in FIG. 2) is shown in FIGS. 3A and 3B by a video processing device 100 or any other suitable device. It can be performed on a video sequence containing text like things. The individual frames 305 are subjected to the procedure outlined in FIG. 2, resulting in the separation of individual text areas such as 310, 315, 360, 365, 370 and 375. It should be noted that the procedure can also be applied to multi-frame integrations integrated to reduce background complexity and increase text clarity. That is, if multiple consecutive frames contain the same text region (and this can be identified if the text regions contain similar signal characteristics such as similar spectral density functions), then multiple Successive frames can be integrated (eg, averaged). This tends to make the text areas clearer and the text better distinguished against the background. If the background is an animation, this procedure necessarily reduces the complexity of the background. It should be noted that some of the advantages of such signal averaging can also be obtained from sources where the time integration is done for motion picture enhancement, as in modern televisions. Thus, for the discussion that follows, "single"
The concept of processing on frames is by no means limited to a single "frame grab", the "frame" on which the image analysis is performed can be a composite of one or more consecutive video frames.

First, the image processor 120 separates the colors of one or more frames of the video image and stores the reduced color image for use in extracting text. In one embodiment of the present invention, image processor 120 uses a Red-Green-Blue (RGB) color space model to separate the red component of pixels. An example of what the text portion of the frame looks like is shown in Figure 4A. The red component is most often the most effective in detecting the white, yellow and black colors used primarily for video text. That is, for superimposed (superimposed) text, the isolated red frame provides sharp, high-contrast edges for common text colors. This method can also be used to extract text that is not really overlaid on the video, but is actually part of the video, such as a movie scene highlighting a bulletin board or street sign. . In such cases, the red frame would not be the best to use. In such cases, the gray scale (alpha channel) will provide the best starting point. It should be noted that in other embodiments of the invention, the image processor 120 may use various color space models, such as grayscale images, or the Y component of YIQ video frames.

The separated frame images are stored in the image text workspace 132.
Then, in step S210, the captured image is sharpened before further processing. For example, the following 3x3 mask can be used: -1 -1 -1 -1 -1 8 -1 -1 -1 -1 In this case, each pixel has 8 times its own negative of each adjacent pixel. It will be added. The above matrix representation for bitmap filters (or "masks") is conventional in the art. There are many such derivative filters known in the art, and the present invention seeks to use any of a variety of different techniques to separate text regions. The above is just a very simple example. The filtering step includes, for example, detecting a gradation along a certain dimension and detecting a gradation along another dimension (simultaneously smoothing in each orthogonal direction).
, Followed by the addition of the results of these two filters, and so on. In step S210, for example, RC Gonzalez and RE in "Digital Image Processing" by Addison-Wesley Publishing Company in 1992.
Random noise can be reduced using a median filter as described by Woods.

Other edge filters can be used for edge detection. Through this filter, edges in a sharpened (red, grayscale, etc.) image can be amplified (preferably amplified), while non-edges are attenuated, for example using the edge mask below. , -1 -1 -1 -1 -1 12 -1 -1 -1 -1 Again, each pixel is the sum of the above coefficients (weights) applied to itself and the adjacent pixels. The result of the above filtering step is illustrated in FIG. 4C. The original image 163 is edge-filtered to obtain a differential image 164, then the image is edge-enhanced to obtain a final image 165, and the final image is subjected to the following filter processing.

[0030]   In step S215, the threshold edge filter or "edge detector"
Applied. Edge_{m, n}Represents m, n pixels of the M × N edge image, and F _{m, n} Represents the enhanced image resulting from step S210, then
The following formula can be used for detecting the error. Formula 1

[Formula 1] Here, 0 <m <M and 0 <n <N, and L _edge is a constant or non-constant threshold value. The value w _{i, j} is the weight from the edge mask. The outermost pixels can be ignored in the edge detection process. Again, it should be noted that a sharpening filter can be implicitly applied to this threshold process.

The edge threshold L _edge is a predetermined threshold and can be a fixed value or a variable value. Using a fixed threshold results in salt and pepper noise and discontinuities in the fixed edges around the text. Known opening processes (eg erosion processes followed by dilation processes) result in the loss of some of the text. Adaptive threshold edge filters, ie filters with variable thresholds, improve on this trend,
Greater improvement over the use of quiescent thresholds.

In step S220, a first fixed threshold is applied using the edge detector in a mode of adjusting the edge detection threshold and then identified in the fixed threshold step. The local threshold is lowered and any pixel adjacent (within a certain tolerance) to the edge pixel is re-applied. In the other mode, the latter effect applies a smoothing function to the result of the threshold step (assuming the result is stored at a pixel depth greater than 2) and then rethresholds thresholding. It can be easily achieved by executing. This causes pixels marked as non-edges to be marked as edges. The degree of threshold reduction for a pixel preferably depends on the number of adjacent pixels marked as edges. The rationale behind this is that if the adjacent pixel is an edge, then the current pixel is likely to be an edge. Edge pixels resulting from local threshold reduction are not used in the reduced threshold calculation for adjacent pixels.

As another example, using a fixed threshold with a low pass weighting function, a single or a small number of non-edge pixels surrounded by strong edge pixels (pixels with high gray levels) are marked as edge pixels. It can also be guaranteed. In fact, all the steps S210 to S220 described above can be described by a single numerical operation on the summation, in the form of equation 1, but with a wider range. Separation into these individual steps should not be considered necessary or limiting and may depend on the specifics of computing equipment and software and other requirements.

Once a character edge is detected, the image processor 120 may perform preliminary edge filtering to remove image regions that either do not contain text or where the text cannot be reliably detected. To execute. For example, a frame with extremely few edges, a very low edge density (number of edge pixels per unit area) or a low set of edge pixels (ie, they do not form long distance structures, eg noise) , Can be excluded from further processing.

The image processor 120 can perform edge filtering at different levels. For example, edge filtering can be performed at the frame level or sub-frame level. At the frame level, the image processor 12
A 0 can ignore a frame if it appears that the larger part of the frame is made up of edges. As another example, a filtering function such as spectral analysis can be applied to determine if the frame is likely to have too many edges. This can result from dense, strong edged objects in the frame. The assumption is that overly complex frames contain a large proportion of non-character details, which is disproportionately expensive to filter via character classification.

When frame-level filtering is used, the image processor 120 maintains an edge counter to determine the number of edge pixels in the image frame. However, this can lead to skipping and ignoring frames containing clear text, such as frames with noisy parts and parts of clear text. To prevent the exclusion of such image frames or image sub-frames,
The image processor 120 can also perform edge filtering at the sub-frame level. To do this, the image processor 120 can divide the frame into smaller regions. To achieve this, the image processor 120 can, for example, divide the frame into three groups of pixel columns and three groups of pixel rows.

The image processor 120 then determines the number of edges in each sub-frame and sets the associated counter accordingly. If the subframe has more than a predetermined number of edges, the processor can discard the subframe. The predetermined maximum edge count value per region can be set according to the amount of time required to process the image region or the probability that the size with respect to the pixel density makes the recognition accuracy lower than the desired minimum value. A large number of subframes can be used to ensure that small areas of clear text surrounded by areas that are identified as uninterpretable are lost.

Next, in step S225, the image processor 120 performs connection component (CC) analysis on the edges generated in the previous step. This analysis aggregates all consecutive edge pixels within a certain tolerance. That is, all edge pixels that are adjacent to or within a distance of another edge pixel are merged with that pixel. Finally, the integration process defines structured or connected components, each having a contiguous or nearly contiguous set of edge pixels. The motivation for this is that each text character region is assumed to correspond to a single CC. What is the above tolerance depending on the resolution of the image capture, the degree of upsampling (percentage of pixels added by interpolation from the original image) or the degree of downsampling (percentage of pixels deleted from the original image) It can be set to a suitable value.

Referring now to FIG. 4B, inadvertent gaps or discontinuities between CCs corresponding to consecutive characters may appear as a result of edge detection with a fixed threshold. For example, interruptions such as those shown at 171 and 172 may occur. The use of the edge detection method described helps to ensure the integration of such interrupted CC parts. Beginning with a discontinuity as in the letters on the left side of FIGS. 5A and 5B, as a result of the CC integration method, the points at discontinuities 174, 175 and 176 are identified as edge points and a single connection configuration as in 181 and 182 It will be integrated into each substructure. It should be noted that closing the "wrong" interruptions in the connected areas can be achieved in various ways in addition to the particular way described above. For example, the expansion treatment can be applied after the erosion treatment or the thinning treatment. To prevent the effect of increasing the total area of the edge pixels, the dilation process can be followed by a thinning process before detecting the connecting component. It is also possible to increase the grayscale depth of the binarized thresholded image as a result of applying Equation 1 and then apply a smoothing function and perform the thresholding (Equation 1) again. . There are many image processing techniques that can be used to achieve the desired closure effect. Yet another example is to mark these pixels as edge pixels when they are substantially surrounded by edge pixels in the form of a continuous sequence such as the one illustrated in FIG. 5C. That is, each of the 24 cases shown is a pixel with 8 pixel neighbors. In each of these cases, the neighbors have 5 or more edge pixels in the form of a continuous sequence. Of course, the numbers in the above sequence can be changed or special cases can be added to the group. Furthermore, the size of the matrix can be increased. The type of pixel that is preferred to be marked as an edge by an algorithm such as that defined for FIG. 5C is such that the pixel is considered unlikely to be part of a continuous discontinuity. Similar results are obtained with the closing process (
Expansion treatment followed by an erosion treatment) or by using a less sharpening treatment in the mask or by using a pretreatment for the threshold treatment (application of equation 1).

A CC is a set of pixels determined to form a continuous sequence without non-edge pixels that divide one part into the other part. For each CC, a list is created that includes the coordinates of the leftmost, rightmost, topmost and bottommost pixels in the structure, along with an indicator of the position of the structure, such as the coordinates of the center of the structure. The number of pixels forming the connection component substructure can also be stored. It should be noted that the pixel count represents the area of a particular connection component. The area of a given connection component to determine a given system and / or user threshold, which connection component should be passed on to the next processing step,
It can be used to define maximum and minimum limits for height and width.
The final step is the filter to determine if the CC is qualified as a character. C is too small to satisfy other heuristics by itself.
It can be used to combine Cs or to split oversized ones.

In step S 230, the image processor 120 rearranges the connection components that meet the criteria in the previous step in ascending order based on the position of the pixel at the bottom left corner. The image processor 120 sorts based on the coordinates of the pixels. The sorted list of connection components is examined to determine which CCs form blocks of text ("boxes").

The image processor 120 assigns the first CC to the first box and as the first or current box for analysis. The image processor 120 uses each subsequent CC
Is checked to see if its bottommost pixel is on the same horizontal line (or the nearest line) as the corresponding pixel in the first CC. That is, each subsequent CC is added to the current text box if its vertical position is close to that of the current CC. If so, it is considered to belong to the same line of text. The vertical coordinate difference threshold can be fixed or variable. Preferably, the closeness of the horizontal coordinates of the second CC is a function of the height of the CC. The horizontal distance of the new addition candidate to the current text box is also determined to see if it falls within an acceptable range.

If the CC does not meet the criteria for merging into the current textbox, a new textbox is generated and the outlying CC is marked as its first component. This process results in multiple text boxes for a single text line in the image. If the next connected component in the sequence has a significantly different vertical coordinate or horizontal coordinate less than that of the last CC, the current text box is closed at the end of the horizontal crossing and the new text box is closed. Will start.

For each box, the image processor 120 then performs a second level integration process on each of the text boxes created by the initial character integration process. This process is erroneously interpreted as a line of another text, thus merging text boxes placed inside another box. This can result from strict connection component integration criteria, or poor edge detection can result in multiple CCs for the same character.

The image processor 120 compares each box with the text box that follows it for a set of conditions. The criteria for two text boxes are: a) The bottom of one box is within a certain vertical spacing of the other, which spacing corresponds to the expected line spacing. Also, the horizontal spacing between the two boxes is
Less than a variable threshold based on the average width of the characters in the first box, b) the center of any of these boxes is located within the area of the other text box, or c) the top of the first box is second Overlapping with the bottom of the text box, the left or right side of one box is within a few pixels of the left or right side of the other, respectively.

If any of the above conditions are met, the image processor 120 deletes the second box from the list of text boxes and merges it into the first box. The image processor 120 repeats the process until all the text boxes have been determined with respect to each other and combined as much as possible.

In step S235, the image processor 120 accepts the text boxes obtained from step 230 as text lines if the text boxes comply with certain constraints of area, width and height. For each of the text boxes, image processor 120 extracts from the original image the sub-image corresponding to that text box. The image processor 120 then binarizes the sub-image in preparation for character recognition. That is, the color depth is reduced to 2 and the thresholding is set to a value that ensures that the characters are properly distinguished from the background. This is a difficult problem and may involve multiple steps such as integrating multiple frames to simplify complex backgrounds.

The threshold for binarizing an image can be determined as follows.
The image processor 120 modifies the textbox image by calculating the average grayscale value (AvgFG) of the pixels in the textbox. This is used as a threshold for binarizing the image. The average gray scale value (AvgBG) of the area around the text box (eg, 5 pixels) is also calculated. The sub-image marks everything above AvgFG as white and everything below AvgFG as black. Then, the average of the pixels marked as white, Avg1, is calculated, and
The average of the pixels marked as black, Avg2, is also calculated.

Once the text box has been converted to a black and white (binary) image, the image processor 120 compares Avg1 and Avg2 with AvgGB. Areas with averages close to AvgBG are assigned as background, other areas as foreground (or text). For example, if the average of the black area is close to AvgBG, then the black area is converted to white and vice versa. This ensures that the text will always have consistent values for input to the OCR program. The image processor 120 then
The extracted frame text is stored in the image text workspace and the process continues with the next frame in process step 205. It should be noted that prior to the local threshold processing, a super resolution step can be performed to improve the text resolution.

Next, the individual character regions must be separated before the classification can be performed. Various heuristics may be used to separate individual character regions from lines of text, such as, for example, character height to width ratios, upper and lower thresholds for height and width. These heuristics typically fall into the prediction of acceptable values for various dimensional features.

The connection component may fail to correspond to a character due to lack of clarity of the original text. Referring now to FIGS. 6A-6D, if CC segmentation fails, other tools can be used for segmenting characters along horizontal lines. An example is the vertical projection 425, which is a function of the horizontal coordinate, and whose value matches the X coordinate and is the number of foreground pixels in the vertical column in the current text box (and possibly the one shown). So it is proportional to the grayscale value). That is, the vertical column in which the pixels are integrated does not exceed the size of the text box, and thus only the characters of the current line are measured. This "greyscale" vertical projection 425 can also be weighted by a window function 425, the width of which is proportional to the expected width for the next character in the sequence. Window function 42
The result of weighting by 5 is shown at 420. The minimum projection value can also be used to define the left and right edges of a character.

Referring to FIG. 7A, the method for separating character regions starts at the first CC and proceeds sequentially through the text box. Starting in step S310, the first,
Alternatively, the next CC is selected. In step S312, the selected CCs are determined to see if they meet them for dimensional heuristics.
A heuristic determination on a CC may indicate that the CC is not likely to be a complete character or that the CC is too large and likely to contain more than one character.
If it is found in step S314 that the CC is too large, another method of segmenting characters, such as the grayscale projection described above, is used in step S316.
Applied in. If the CC is found to be too small in step S322, then the next CC is determined for the heuristic in step S318. If the result shows in step S320 that the following CC is also too small, the flow returns to step S310 until the current and subsequent CCs have been merged in step S326 and all character regions have been separated. If the following CC is not too small, then the current CC is discarded in step S324 and flow returns to step S310.

Referring to FIG. 7B, another method of segmenting characters attempts to categorize these alternatives by evacuating alternative character regions that have failed heuristics. In the classification, the alternative that achieves the highest confidence level is selected. In this case, the other character areas are processed accordingly. For example, if the images corresponding to the two integrated CCs were classified with a high confidence measure, the field corresponding to the CC with the first CC integrated is no longer treated as a separate character field. In step S330, the first or next CC is selected. In step S332, the selected CC is
It is determined to see if the CC meets them for dimensional heuristics. If it is found in step S334 that the CC is too large, another method of segmenting characters is applied in step S336. If it is found in step S338 that the CC is too small, the current CC and the current CC combined with the next CC are both held as alternative character fields. When these character fields are sent for classification as described below, the confidence measure is used to make a choice between these alternatives. The flow returns to step S330 until all character areas have been separated. If the guttering process of step S336 results in a low confidence measure, the oversized and divided fields are retained as alternatives for classification and the classification results are used for selection between these alternatives. .

The area corresponding to a character need not be defined as a linear box. These areas are rubber band boundary areas (convex polygons with an arbitrary number of sides), or orthogonal convex linear polygons (all horizontal and vertical line segments connecting two internal points are It may be a straight polygon such that it lies completely inside), or any other suitable shape that substantially encloses the important features of the predicted symbol or character.

It should be noted that the textbox information can be deleted altogether and the connection component can be used directly to identify the candidate character area. However, in such cases, a greater number of connection components
It is expected that these components will fall outside the particular symbol set to be mapped (classified). It should also be noted from the above description that it is clear that the techniques described above are broadly applicable to the classification of symbols and are not limited to the classification of textual characters.

Referring to FIG. 8, once all the character regions have been separated (included by step S405), the characters can be sorted in order. Next, step S410.
At, the first or sequential character area is selected. In step S415, some suitable image analysis is performed on a portion of the original image (or its red portion) in preparation for feature analysis. For example, the image is binarized (thresholded),
It can be grayscale imaged, binarized and refined, and so on.
The preprocessing varies depending on the feature space used.

With reference to FIGS. 9A-9D, for example, the feature space may utilize certain feature points (as described below). The minutiae can be identified using skeleton characters, and the image can be binarized (FIG. 9B) and then thinned (FIG. 9C) to derive them from ordinary video characters (FIG. 9A). You can The feature points (465 to 468 in FIG. 9D) can then be derived as corner points 465, bends 466, intersections 467 and endpoints 468 of the thinned characters 460, 470. This type of image processing is suitable for the angle histogram feature space described below.
Only a low degree of image processing is required to calculate the size invariant moment. It should be noted that other minutiae definition systems can be used as well.

Referring again to FIG. 8, the original character can be subjected to various different analyzes to define a feature vector, which is the input of a properly trained backpropagation neural network (BPNN). Can be supplied to the edge. For techniques that use size-invariant moments, unthinned or thinned characters can be used. In step S420, the selected feature vector is generated by appropriate image analysis. A variety of these can be used. A number of different feature spaces have been defined for the applications to which this patent pertains.
The defined feature space detailed below is size and rotation invariant and is considered to be particularly suitable for video character classification using the BPNN classifier.

A first feature space is derived from the feature points of the thinned characters, as shown by FIGS. 9A-9D. 10A and 10B, first, Delaunay triangulation (FIG. 10A) or Voronoi diagram (Voronoy diag)
ram: FIG. 10B) is derived from the feature points 12. Image processor 120 performs the triangulation described above and then generates a list of interior angles for each triangle 1-6. The image processor then uses this inventory to generate an angle histogram as shown in FIG. 11A. The histogram simply represents the frequency of angles A, B and C in a given size range in the set of triangles 1-6 defined by the above triangulation. Note that other triangulation methods or polygon generation methods can also be used. For example, referring to FIG. 10B, a set of Voronoi polygons 17 and 18 can be used to define a set of angles A ′, B ′ and C ′ each associated with a vertex 14 of the Voronoi diagram. . The resulting angle histogram acts as a feature vector for the particular character from which the feature points were derived.

Other size and rotation invariant features can be added to the feature space, such as the number of horizontal lines, the number of intersections, the number of end points, holes, inflection points and midpoints. Another variation on the angle histogram is the use of only the two largest (or smallest) interior angles of each triangle. Yet another variation of the angle histogram is to use a two-dimensional angle histogram instead of a one-dimensional angle histogram. For example, referring to FIG. 11B, the largest (or smallest) pair of angles for each triangle is aligned (or aligned by size) for each triangle in Delaunay triangulation (or for each vertex in the Voronoi diagram). Define a pair. The first element in each aligned pair is used for the first dimension of the matrix and the second element is used for the second dimension of the matrix. In this way, the association between angles is preserved as information for training and classification with the BPNN classifier.

Yet another feature space that may be particularly suitable for video character BPNN classifiers is an array of size invariant moments. These moments are defined by the following equations. There are many distinct moments that can be used in that situation, but a certain minority is chosen for this application.
First, the pixel index of the pixel position corresponding to the center of the mass i bar and j bar is

[Formula 2] Where B [i] [j] is 1 if the i, jth pixel of the thresholded image is a foreground pixel, and 0 otherwise. Is the total area of the foreground pixels given by

[Formula 3] The translation invariant moment is

[Formula 4] Where M _{p, q} is _{p, q} of the character image given by the following equation:
It is the th raw moment.

[Formula 5] The invariant moment selected for input to the BPNN is

[Formula 6] Is.

Referring again to FIG. 8, in step S425, each feature vector is provided to the trained BPNN, which is a very strong candidate responsive to various candidate classifications and, optionally, inputs. Output candidates and. B if there are multiple candidate characters
The best estimate is made in step S430 by combining the probability output by the PNN and the frequency of the usage data for the estimated language and context. Such data can be gathered from different types of material such as, for example, transcripts of television advertisements, printed matter, internet streaming or downloaded files. One way to combine is to weight the probabilities output by the BPNN by the corresponding probabilities associated with the usage statistics.

For the person skilled in the art, the present invention is not limited to the details of the illustrative embodiments described above, and the invention can be embodied in other specific forms without departing from its spirit or essential attributes. It will be clear that it can also be implemented. For example, the text analysis presented above described preferences for horizontally aligned text. Obviously, the same method can be applied to other alignments such as vertically aligned text, text along a curve, etc.

Accordingly, the above embodiments should be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than the foregoing description, and the claims All changes that come within the meaning and range of equivalency are therefore intended to be covered by the appended claims.

[Brief description of drawings]

FIG. 1 is a block diagram showing an apparatus that can be used to implement the present invention.

[Fig. 2]   FIG. 2 is a flowchart showing a character classification method according to an embodiment of the present invention.

FIG. 3A illustrates a text area on a video screen containing information that can be classified according to one embodiment of the present invention.

FIG. 3B also shows a text area on a video screen containing information that can be classified according to one embodiment of the present invention.

FIG. 4A shows the appearance of a text segment from a captured digital image of a video frame.

FIG. 4B   FIG. 4B shows the text segment after edge detection filtering.

FIG. 4C shows the effect of several stages of filtering during or before edge detection, which are presented to illustrate the concepts associated with the present invention. , Please note that it does not really show the intermediate results.

FIG. 5A   FIG. 5A shows the effect of edge filtering according to an embodiment of the present invention.

FIG. 5B   FIG. 5B also shows the effect of the edge filtering process according to the embodiment of the present invention.

FIG. 5C shows an example of a gap closing algorithm that can be used in the present invention.

FIG. 6A illustrates a technique for segmenting a text line according to one embodiment of the invention.

FIG. 6B also illustrates a technique of text line segmentation according to one embodiment of the present invention.

FIG. 6C also illustrates a technique for segmenting a text line according to one embodiment of the present invention.

FIG. 6D also illustrates a technique for segmenting text lines according to one embodiment of the invention.

FIG. 7A is a flowchart showing a technique of creating and managing a connection constituent part by a filtering process according to an embodiment of the present invention.

FIG. 7B is also a flowchart showing a technique of creating and managing a connection constituent part by filtering processing according to an embodiment of the present invention.

[Figure 8]   FIG. 8 is a flowchart showing a character classification method according to an embodiment of the present invention.

FIG. 9A shows segmented character filtering for deriving feature vector precursors.

FIG. 9B also shows segmented character filtering to derive feature vector precursors.

FIG. 9C also shows segmented character filtering to derive feature vector precursors.

FIG. 9D also shows segmented character filtering to derive feature vector precursors.

FIG. 10A illustrates Delaunay triangulation steps in the image filtering step of the character classification process according to one embodiment of the present invention.

FIG. 10B shows Voronoi diagram steps in the image filtering step of the character classification process according to one embodiment of the present invention.

FIG. 11A   FIG. 11A illustrates an angle histogram type feature space according to an embodiment of the present invention.

FIG. 11B   FIG. 11B also shows an angle histogram type feature space according to an embodiment of the present invention.

[Explanation of symbols]

  100 ... Image text analysis system   110 ... Video processing device   120 ... Image processor   130 ... RAM   132 ... Image text workspace   134 ... Text analysis controller   140 ... Storage unit   150 ... User I / O card   160 ... Video card   170 ... I / O buffer   175 ... Processor bus   180 ... video source   185 ... Monitor

─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Dimitrova Nebenka             Netherlands 5656 aer ind             Fenprof Holstraan 6 F-term (reference) 5B029 AA02 BB02 CC28 DD05 EE08

Claims (7)

[Claims] 1. A method for classifying symbols in an image data stream containing symbols, the method comprising: training a backpropagation neural network (BPNN) into a feature space containing at least two shape-dependent features; For capturing an image from a stream, detecting an image region extending in the same space as a symbol to be classified embedded in the video data stream, obtaining a feature vector from the image based on the feature space, and classifying the symbol And providing the feature vector to the BPNN. 2. The method of claim 1, wherein the at least two shape dependent features include size, translation and rotation invariant moments. 3. The method of claim 1, further comprising identifying feature points in the image, wherein the at least two shape dependent features are incident angle magnitudes that appear in a triangulation of the feature points. A method that includes 4. The method of claim 1, further comprising identifying feature points in the image and forming at least one of a Delaunay triangulation and a Voronoi diagram based on the feature points, The method wherein the at least two shape dependent features include a histogram representing angles of incidence that appear in at least one of the Delaunay triangulation and Voronoi diagram. 5. An apparatus for classifying symbols in an image data stream containing symbols, the image data storage unit comprising an input section and an output section connected to capture data from the image data stream, and An image processor connected to the output of the image data storage unit and programmed to detect an image spread in the same space as the symbol to be classified embedded in the image data stream, Includes a backpropagation neural network (BPNN) trained in a feature space, the feature space includes at least one shape-dependent feature, the image processor obtains a feature vector from the image based on the feature space, and Programmed to provide the feature vector to the BPNN to classify symbols, Location. 6. The apparatus according to claim 5, wherein the image processor identifies feature points in the image and forms at least one of Delaunay triangulation and Voronoi diagram based on the feature points. Wherein the derivation of the feature points comprises refining the image in binary form, wherein the image processor has at least two shape dependent features in at least one of the Delaunay triangulation and Voronoi diagram. A device further programmed to include a histogram representing the incident angles of incidence. 7. A computer program product for an image data processing device, the computer program product causing the image data processing device to perform the method of claim 1 when loaded into the image data processing device. A computer program product having a set of instructions.