EP0543599A2 - Procédé et appareil de détection de marquage à main d'images - Google Patents

Procédé et appareil de détection de marquage à main d'images Download PDF

Info

Publication number
EP0543599A2
EP0543599A2 EP92310438A EP92310438A EP0543599A2 EP 0543599 A2 EP0543599 A2 EP 0543599A2 EP 92310438 A EP92310438 A EP 92310438A EP 92310438 A EP92310438 A EP 92310438A EP 0543599 A2 EP0543599 A2 EP 0543599A2
Authority
EP
European Patent Office
Prior art keywords
image
hand
document
marks
regions
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP92310438A
Other languages
German (de)
English (en)
Other versions
EP0543599B1 (fr
EP0543599A3 (en
Inventor
Dan S. Bloomberg
Dasari Lakshmi
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xerox Corp
Original Assignee
Xerox Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xerox Corp filed Critical Xerox Corp
Publication of EP0543599A2 publication Critical patent/EP0543599A2/fr
Publication of EP0543599A3 publication Critical patent/EP0543599A3/en
Application granted granted Critical
Publication of EP0543599B1 publication Critical patent/EP0543599B1/fr
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/14—Image acquisition
    • G06V30/1444—Selective acquisition, locating or processing of specific regions, e.g. highlighted text, fiducial marks or predetermined fields
    • G06V30/1448—Selective acquisition, locating or processing of specific regions, e.g. highlighted text, fiducial marks or predetermined fields based on markings or identifiers characterising the document or the area
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition

Definitions

  • This invention relates generally to image processing and more specifically to a morphological technique and apparatus for: discriminating between regions of a document which have been hand marked with an ordinary pen or other ordinary writing utensil and, regions of a document which have not been marked; and extracting a desired region.
  • regions may be marked for the purposes of: identifying text to be sent to an optical character recognition (OCR) system, for the purpose of retrieval of the ASCII representation and identification of fields or key words for database filing; identifying parts of an image that are not to be sent to an optical character recognition system; identifying parts of an image that are to be stored as a bitmap image; and identifying a region of a form for which some action is to be taken.
  • OCR optical character recognition
  • U.S. Patent No. 5,029,224 to Fujisawa describes a marked region recognition apparatus which recognizes an arbitrary marked region of a document image from a mark signal which indicates whether or not there exists a mark for delineating the marked region.
  • the marked region recognition apparatus comprises a first storing part for storing a mark signal for at least one scanning line, a second storing part for storing a coordinate in a main scanning direction where the mark region ends for each scanning line based on the mark signal stored in the first storing part, and a recognition part for recognizing an inside and an outside of the marked region and producing a marked region signal which indicates the inside or the outside of the marked region for a present scanning line contiguous with the marked region signal of a previous scanning line, where a state of the marked region signal of a previous scanning line is obtained from the first and second storing parts.
  • U.S. Patent No. 4,016,906 to Matsunawa et al. describes an apparatus capable of detecting color marking on a multicolored document, then performing specific image processing on the inside or outside of the region designated by the color marker.
  • a region extraction circuit detects a region marked by a specific color marker by sending a pulse when the marker color is detected during a scan. The duration between pulses thus provides the width of the marked region.
  • U.S. Patent No. 4,720,750 to Watanabe describes an image forming apparatus for an electronic copying machine which is capable of designating and erasing any part of an original document.
  • a masking area is drawn on an area designation sheet which is then read and stored by the copying machine.
  • the original document is then placed on the copying machine and the marked/mask area is erased, i.e., not copied, in accordance with the stored mask specification from the area designation sheet.
  • U.S. Patent No. 4,908,716 to Sakano describes an image processing apparatus in which an area of a document is designated by a marking entered on the document and a portion encircled by the marking being treated as a marked area which is the subject of a trimming or a masking process.
  • a color felt pen or the like is used to encircle a designated area of a document.
  • a mark detection circuit can detect the marking by detecting the tone of the image.
  • the disparate reflectivity or tone of the marker pen allows marker area detection, whereupon, the marked area can be erased or maintained as desired.
  • the present invention provides a method for processing a scanned first image in a digital computer for differentiating machine marks from hand marks and identifying a location of non-transparent hand marks and hand marked regions, includes the steps of: identifying and differentiating markings on the scanned first image as hand and machine marks using characteristics of the markings, said characteristics including horizontal, vertical, oblique, curved and irregular shapes; identifying regions of the scanned first image to which the hand marks refer; and reproducing the identified regions of the scanned first image to which the hand marks refer without interference from the hand marks.
  • step (a) further comprises steps of reducing the first image by two times, using a threshold of 1; and taking a UNION of the first set of structuring elements with the second image, forming a third image.
  • some of the bounding boxes having less than a predetermined size are small bounding boxes, wherein the small bounding boxes are eliminated.
  • Another aspect of the invention is the provision of a semitopological method for processing an image in a digital computer for extracting regions of a document image encircled by non-transparent hand marks, characterised by
  • the first predetermined size may be a size less than a size of a character or at least as large as a size of a character.
  • a method for processing a document image in a digital computer for identification in the document image of hand drawn lines comprising the steps of:
  • step (a) is preceded by a step of reducing the document image and/or includes using at least one horizontal or vertical structuring element so that horizontal or vertical lines are identified.
  • a method for processing a document in a digital computer for identifying hand drawn encircled regions of the document image comprising the steps of:
  • step (a) is preceded by by a step of reducing the document image and/or includes OPENING the image using horizontal or vertical structuring elements.
  • the method includes finding bounding boxes of the encircled regions; and extracting the encircled regions from the original document image so as to identify contents of the encircled regions.
  • the method may also include XORing a result of step (d) with a result of step (f) so as to extract the contents of the circled regions.
  • an apparatus for processing a scanned first image in a digital computer for differentiating machine marks from hand marks and identifying a location of non-transparent hand marks and hand marked regions characterised by image scanning means for scanning a first image; processor means for identifying and differentiating markings on the scanned first image as hand and machine marks using characteristics of the markings, said characteristics including horizontal, vertical, oblique, curved and irregular shapes; identification means for identifying regions of the scanned first image to which the hand marks refer; reducing means for reproducing the identified regions of the scanned first image to which the hand marks refer without interference from the hand marks; and an output means for outputting the reproduction of the identified regions.
  • a method and apparatus for image markup detection of hand marks which work on binary scanned images and which utilize binary morphological image processing techniques to expedite the detection process.
  • One of the advantages provided by embodiments of the invention is the provision of methods and apparatus for image markup detection of hand marks which do not require extraneous detection circuitry to operate properly.
  • image refers to a representation of a two-dimensional data structure composed of pixels.
  • a binary image is an image where a given pixel is either "ON" of "OFF".
  • Binary images are manipulated according to a number of operations wherein one or more source images are mapped onto a destination image. The results of such operations are generally referred to as images.
  • the image that is the starting point of processing will sometimes be referred to as the original image or source image.
  • a “morphological operation” refers to an operation on a pixelmap image (a "source” image), that uses a local rule at each pixel to create another pixelmap image, the "destination” image. This rule depends both on the type of the desired operation to be performed as well as on the chosen "structuring element”.
  • Pixels are defined to be ON if they are black and OFF if they are white. It should be noted that the designation of black as ON and white as OFF reflects the fact that most documents of interest have a black foreground and a white background. The techniques of the present invention could be applied to negative images as well. The discussion will be in terms of black on white, but the references to ON or OFF apply equally well to images which have been inverted and, therefore, the roles of these two states are reversed. In some cases the discussion makes reference to a "don't care" pixel which may be either an ON or an OFF pixel.
  • a “structuring element” refers to an image object of typically (but not necessarily) small size and simple shape that probes the source image and extracts various types of information from it via the chosen morphological operation.
  • a solid circle is a "hit”, and an open circle is a "miss”.
  • the center position is denoted by a cross. Squares that have neither solid nor open circles are “don't cares”; their value in the image (ON or OFF) is not probed.
  • a binary SE is used to probe binary images in a binary morphological operation that operates on binary input images and creates an output binary image.
  • the SE is defined by a center location and a number of pixel locations, each normally having a defined value (ON or OFF).
  • a “solid” SE refers to an SE having a periphery within which all pixels are ON.
  • a solid 2x2 SE is a 2x2 square of ON pixels.
  • a solid SE need not be rectangular.
  • a horizontal SE is generally one row of ON pixels and a vertical SE is generally one column of ON pixels of selected size.
  • a "hit-miss" SE refers to an SE that specifies at least one ON pixel and at least one OFF pixel.
  • AND, OR and XOR are logical operations carried out between two images on a pixel-by-pixel basis.
  • NOT is a logical operation carried out on a single image on a pixel-by-pixel basis.
  • EXPANSION is scale operation characterized by a scale factor N, wherein each pixel in a source image becomes an N x N square of pixels, all having the same value as the original pixel.
  • REDUCTION is a scale operation characterized by a scale factor N in a threshold level M.
  • the value for the pixel in the destination image is determined by the threshold level M, which is a number between 1 and N2. If the number of ON pixels in the pixel square is greater or equal to M, the destination pixel is ON, otherwise it is OFF.
  • EROSION is a morphological operation wherein a given pixel in the destination image is turned ON if and only if the result of superimposing the SE center on the corresponding pixel location in the source image results in a match between all ON and OFF pixels in the SE and the underlying pixels in the source image.
  • An EROSION will give one pixel in the destination image for every match. That is, at each pixel, it outputs 1 if the SE (shifted and centered at that pixel) is totally contained inside the original image foreground, and outputs 0 otherwise.
  • EROSION usually refers to operations using a structuring element(s) with only hits, and more generally, matching operations with both hits and misses (often called a "hit-miss transform".
  • the term EROSION is used herein to include matching operations with both hits and misses, thus the hit-miss transform is the particular type of EROSION used herein.
  • DILATION is a morphological operation wherein a given pixel in the source image being ON causes the SE to be written into the destination image with the SE center at the corresponding location in the destination image.
  • the Ses used for DILATION typically have no OFF pixels.
  • the DILATION draws the SE as a set of pixels in the destination image for each pixel in the source image.
  • the output image is the union of all shifted versions of the SE translated at all 1-pixels of the original image.
  • a “seed fill” is an operation taking as input two images, and generating a third image as the result.
  • One of the input images is the “seed”, which may be composed of a single ON pixel or of many ON pixels.
  • the other input image is the "mask”, which is typically composed of more than one image component.
  • the two images are aligned.
  • the result of the seed fill is to produce an image that has only those image components in which at least one seed pixel was present in the seed image.
  • the result image is formed by starting with the seed pixels and growing each image regions until it has filled the corresponding image component in the mask.
  • “FillClip” is a morphological operation where one image is used as a seed and is grown morphologically, clipping it at each growth step to the second image.
  • a fillClip could include a DILATION followed by logically ANDing the DILATION result with another image.
  • OPENING is a morphological operation that uses an image and a structuring element and consists of an EROSION followed by a DILATION. The result is to replicate the structuring element in the destination image for each match in the source image.
  • CLOSING is a morphological operation using an image and a structuring element. It includes a DILATION followed by an EROSION of the image by a structuring element.
  • a CLOSE of an image is equivalent to the bit inverse of an OPEN on the (bit inverse) background.
  • An UNION is a bitwise OR between two images.
  • An "intersection” is a bitwise AND between two images.
  • BURRING is a DILATION using a structuring element(s) composed of two or more hits.
  • a “mask” refers to an image, normally derived from an original or source image, that contains substantially solid regions of ON pixel corresponding to regions of interest in the original image.
  • the mask may also contain regions of ON pixels that do not correspond to regions of interest.
  • Text refers to portions of a document or image which comprises letters, numbers, or other language symbols including non-alphabetic linguistic characters such as ideograms and syllabry in the oriental languages.
  • references to DILATION may be in terms of DILATING the image or the image being DILATED (verb forms) or the image being subjected to a DILATION operation (adjective form). No difference in meaning is intended.
  • Morphological operations have several specific properties that simplify their use in the design of appropriate algorithms. First, they are translationally invariant. A sideway shift of the image before transforming does not change the result, except to shift the result as well. Operations that are translationally in variant can be implemented with a high degree of parallelism, in that each point in the image is treated using the same rule. In addition, morphological operations satisfy two properties that make it easy to visualize their geometrical behavior. First, EROSION, DILATION, OPEN and CLOSE are "increasing", which means that if image 1 is contained in image 2, then any of these morphological operations on image 1 will also be contained in the morphological operation on image 2. Second, a CLOSE is extensive and OPEN is antiextensive.
  • image unit means an identifiable segment of an image such as a word, number, character, glyph or other unit that can be extracted reliably and have an underlying linguistic structure.
  • One problem addressed by the invention is identifying regions (i.e., image segments) on a page that have been marked by hand with an ordinary pen (or pencil).
  • the markings can consist of horizontal or vertical lines, or of "circular" marks (either open lines, segments of curved lines, or combinations of the two). Since all markings are by hand, the straight lines will not be the same straightness or smoothness of machine-printed rules, or hand markings using a straight-edge.
  • the image interpretation problem can be broken into several sub-problems
  • a method for finding word boxes or bounding boxes around image units is to close the image with a horizontal SE that joins characters but not words, followed by an operation that labels the bounding boxes of the connected image components (which in this case are words).
  • the process can be greatly accelerated by using 1 or more threshold reductions (with threshold value 1), that have the effect both of reducing the image and of closing the spacing between the characters.
  • the threshold reduction(s) are typically followed by closing with a small horizontal SE.
  • the connected components labeling operation is also done at the reduced scale, and the results are scaled up to full size.
  • the disadvantage of operating at reduced scale is that the word bounding boxes are only approximate; however, for many applications the accuracy is sufficient.
  • the method described above works fairly well for arbitrary test fonts, but in extreme cases, such as large fixed width fonts that have large inter-character separation of small variable width fonts that have small inter-word separation, mistakes can occur.
  • the most robust method chooses a SE for closing based on a measurement of specific image characteristics. This requires adding the following two steps:
  • Sub-problem (1) of the image interpretation problem involves identifying the markings.
  • Several salient characteristics of the hand markings can be used to identify the markings. The characteristics include:
  • the document consists only of text, without rules or line graphics, then it is not necessary to distinguish between hand markings and machine lines, and a probing of the image based on the length of the straight line segments is adequate to separate the hand markings from text.
  • Probing is typically done morphologically, optionally with reduction beforehand, using either an EROSION or an OPENING).
  • Sub-problem (2) of the image interpretation problem provides for identifying regions of the document image to which the handwritten marks refer.
  • the handwritten marks identified in sub-problem (1) are further processed to identify a target part of the document image.
  • a fill operation starting with the identified segments as "seeds", and filling into the connected component of the image of which the identified segments are a part, will provide the connected hand marking.
  • This marking can be an underline, a sideline, a circle, etc.
  • the asperity ratio (width to width) of the bounding box of the (filled) connected component can then be compared with thresholds to determine if the marks are underlines (large width/height), sidelines (large height/width), or circles (both width and height are larger than a minimum threshold value).
  • Underlines typically refer to the image units directly above them; image unit segmentation with association of the neighboring underline is appropriate.
  • the document image can be horizontally CLOSED (or DILATED) so that the letters within the image unit are merged; thus, when extracting by connected components, the entire image unit is obtained.
  • Sidelines typically refer to a block of the document image to the right of the sideline if the sideline is on the left side of the image, and vice versa.
  • Circles typically refer to a part of the document image that is encircled. Encircled means any marking that is intended to demarcate by enclosure, or near enclosure. The circle need not be closed, since the demarcated region, including the hand marking, may be determined by several methods as follows.
  • Sub-problem (3) of the image interpretation provides for reproducing demarcated regions without interference from the hand markings. Problems can arise when the hand marking is connected to some of the material that is to be extracted. Due to such connections, it is generally not possible to remove only the pixels produced by the hand marking from the extracted image. Nevertheless, solutions that work reasonably well do exist.
  • One method according to the invention is extracting those pixels that are believed to constitute the hand marking, and to XOR or set SUBTRACT them from the extracted image. If boundary noise pixels on the hand markings are not extracted, then these will remain in the image after the XOR or set SUBTRACTION; consequently, an OPENING with a structuring element needs to be used to remove them.
  • underlines are typically applied to image units that represent words, so the processing that is intended to lift the material above the underline should be word-oriented. After the image unit is extracted, the pixels representing the underline can be subtracted.
  • Circled regions are the most varied. Assume now that the demarcated region, including the hand markings, have been lifted out by the methods above. If the clipped region is an isolated part of the document image not touched by the circle, the entire region can be extracted, followed by XORing or SUBTRACTING the pixels of the circle component. If the hand marking touches machine printed text, the following options are available:
  • the bitmap of each lifted region should be processed independently. Border pixels may need to be added to avoid boundary effects when CLOSING an image. By processing each region independently, the regions cannot interfere with each other (for example, a large CLOSING will not bridge two different regions). Further, the structuring elements chosen to process a region can be tailored to the dimensions and characteristics of each region, as in the above method for extracting regions demarcated by open circles.
  • the methods and apparatus of the present invention disclose how regions of a document hand marked with an ordinary pen or pencil can be identified, even if marked through lines of text.
  • Two characteristics of markup lines can be used in identifying and extracting those markup lines from a document image.
  • the lines can be identified as being long lines and second, the markup lines can locally appear to be composed of straight line segments whereas text for example has local regions of high curvature.
  • a direct approach to the identification of markup lines, and subsequent processing of hand marked regions extracts those parts of a document image that are composed locally of long straight lines, effectively removing the markup lines from the document image.
  • the direct approach method is shown in a first preferred embodiment in Figure 1 where a first image of a document, as shown for example in Figure 5, is scanned in step 2.
  • the first image of Figure 5 represents a scaled image at 150 pixels/inch.
  • the first image is threshold reduced 2X in step 4 using a threshold of 2 to obtain a first image scaled at 75 pixels/inch.
  • a morphological UNION of OPENINGS on the reduced first image of Figure 5 is taken in step 6, using a set of eight structuring elements as shown in Figures 27A-H, and forming a second image as shown in Figure 17.
  • the structuring elements of Figure 27A-H each have a length 9 and as a set they represent eight lines in different orientations.
  • any pixels that lie within a part of the reduced first image that entirely cover at least one structuring element pattern are accepted.
  • the result in Figure 17 is quite clean with strong markup lines.
  • the bounding boxes are created as shown in step 8 and filled, forming a mask of the markup region as shown in Figure 20.
  • Figure 2 shows a second preferred embodiment of an image markup method using the direct approach to the identification and extraction of hand marked regions.
  • the first three steps of the method are identical to steps 2,4 and 6 in Figure 1.
  • Those first three steps include scanning a first image (from Figure 5) in step 2, reducing the image in step 4, and taking a UNION of OPENINGS of the first image using the set of eight structuring elements shown in Figures 27A-H in step 6.
  • the departure from Figure 1 occurs in step 16 of Figure 2 where the second image shown in Figure 17 is reduced by 2X to 38 pixels/inch, using a threshold value of 1.
  • the second image is processed by taking a UNION of the same set of OPENINGS used in step 6, i.e., the set of structuring elements shown in Figures 27A-H, to form a third image as shown in Figure 18. Since the second preferred embodiment occurs at a 2X reduction compared with the reduction in step 4 of the first embodiment, the structuring elements are effectively twice as long as they were in the prior step 4 UNION. As seen from Figure 18, the result of the morphological operations in providing the third image is a very clean image, effectively showing only markup lines.
  • the third image is next reduced by 4X in step 22 by using two 2X reductions, each with threshold 1, creating a fourth image with a resolution of 9.5 pixels/inch as shown in Figure 19.
  • a bounding box fill of the fourth image in step 24 results in a fifth image (mask) as shown in Figure 20. Small bounding boxes may be eliminated in step 26, providing error control for stray marks and extremely small enclosures that would not likely be purposely marked.
  • Figure 28 deskews the original image shown.
  • the deskew procedure is optional, but it helps when using the handLineDiscrimination procedure.
  • a 4X reduction of the first image of Figure 28 is provided and shown in Fig. 29 (although a 2X reduction is also acceptable).
  • the reduction procedure is optional, but it helps efficiency, since many of the subsequent operations can be performed much faster on a reduced image.
  • horizontal lines have been extracted with an OPENING (or less satisfactorily an EROSION) using a long horizontal structuring element.
  • the structuring element used in this case was 20 pixels long, which is reasonable for images with resolution of 40-80 pixels/inch which corresponds to a reduction between 4X and 8X from typical scanning resolution of 300 pixels/inch.
  • the hand line is weak, i.e., thin, and slanted, it may not be possible to extract a long horizontal line.
  • a small vertical dilation can be done before the horizontal OPENING to make the lines thicker.
  • another DILATION can be done after horizontal line extraction to join slightly separated components.
  • the connected components have been found and short lines have been removed using any number of conventional techniques.
  • Figure 31 shows bounding boxes for those components remaining after the short lines have been culled.
  • handwritten lines are selected. Possible discriminating factors (both used in this example) include the asperity ratio (width to height) and the fraction of the bounding box containing ON pixels.
  • the handwritten marks are extracted at full resolution as follows.
  • a general approach to the method of extracting encircled regions of a document image which have been hand marked with an ordinary pen or pencil (excluding a highlighter pen) is illustrated in the flowchart of Figure 46.
  • An image is scanned in step 200 and the resultant document image A is deskewed in step 202.
  • the deskewed image B is then OPENed in step 204 using at least one horizontal or one vertical structuring element.
  • the OPENed image C then has either horizontal or vertical image segments according to the type of structuring element used in step 204.
  • Bounding boxes are determined about image units in step 206 resulting in a data structure D which includes bounding rectangles of horizontal or vertical line segments.
  • step 208 bounding boxes are removed from the data structure D which include only machine markings, resulting in a data structure E.
  • the regions represented by the data structure E in step 210 are extracted from the deskewed image B, resulting in an image F of hand drawn line segments.
  • the hand drawn line segments of image F are used as a seed in step 212 for a seed fill operation using image B as a clipping mask.
  • the result of the seed fill operation is an image G of filled circles representing the hand drawn marks.
  • step 214 bounding boxes are found around the filled circles of image G, resulting in a data structure H which includes bounding rectangles of hand drawn marks.
  • the regions represented by the data structure H are then the extracted from the deskewed image B in step 216, resulting in an image J containing hand drawn marks and contents.
  • image J (containing hand drawn marks and contents) is exclusive-ORed with image G.
  • the result of the XOR is an image containing only contents of hand drawn regions, i.e., no hand drawn marks remain in the
  • An indirect approach to the identification of hand markup lines involves removing all print from the document image except the markup lines.
  • the indirect approach shown by the steps in Figure 3 begins in step 30 by scanning a first image, shown in Figure 5, at 150 pixel per inch.
  • the scanned first image of Figure 5 can be created, for example, by scanning at 150 pixels/inch or by scanning at 300 pixels/inch then reducing by a factor of 2 with a threshold of 2.
  • the second image shown in Figure 6 is formed by CLOSING the first image of Figure 5 with a first horizontal structuring element such as the structuring element of length 8 shown in Figure 21.
  • Other structuring element orientations can also be used, but the horizontal orientation is most effective in CLOSING up curves and spaces between characters.
  • the CLOSING operation in step 32 identifies regions of high curvature in the first image.
  • the result of the CLOSING operation of step 32 ( Figure 6) and the first image of step 30 ( Figure 5), are XORed at the XOR gate 34 shown in Figure 3.
  • the result of the XOR is shown in Figure 7 as a third image which includes only those pixels that were added by the CLOSING operation of step 32. Most of those pixels are expected to be near lines that have high curvature since lines with low curvature should not generate many such pixels.
  • a fillClip operation on the third image of Figure 7 is segmented into two steps in Figure 3, namely DILATING the third image in block 36 followed by logically ANDing the result of the DILATION with the first image in step 38.
  • the third image of Figure 7 is DILATED in step 36 with a solid 3 x 3 structuring element.
  • the DILATED third image is then ANDed with the first image in step 38 forming the fourth image of Figure 8.
  • the fillClip process fills back the lines of the document that generated the high curvature pixels in Figure 6. In order to completely fill back the lines of the document as shown in Figure 8, the fillClip operation can be iterated more than once.
  • the third image of Figure 7 could be sequentially DILATED with a solid 3 x 3 structuring element, thence logically ANDing the result of the DILATION with the first image of Figure 5 to produce a fourth image as shown in Figure 8.
  • the fourth image of Figure 8 output from AND gate 38, is then XORed in gate 40 with the first image of Figure 5 from step 30, forming a sixth image as shown in Figure 10.
  • many alternative morphological processes can be used to obtain the fifth image shown in Figure 9.
  • the method chosen in this embodiment includes taking an UNION in step 42 of the fifth image of Figure 9 with a set of openings, specifically using a set of four structuring elements as shown in Figures 23-26.
  • the morphological operation involves OPENING the fifth image of Figure 9 by each of the structuring elements in turn, then taking a UNION of the results, forming a sixth image shown in Figure 10 with reduced noise, observed as containing generally thin vertical lines, wherein every pixel in Figure 10 belongs to a run of 5 pixels in at least one of the four orientations of the structuring elements shown in Figure 23-26.
  • the background noise is further reduced by OPENING the sixth image of Figure 10 in step 44 with a second horizontal structuring element of length 2 as shown in Figure 22, forming the seventh image of Figure 11.
  • the background noise is sufficiently reduced to allow a number of alternatives for continued image processing, such as CLOSING the seventh image with a sequence of structuring elements to close up the small gaps in the markup lines, followed by small openings to remove the background noise.
  • the seventh image is reduced by a factor of 4, i.e., from 150 pixels/inch to 38 pixels/inch, using two threshold reductions of 2X with threshold 1 for each reduction, producing an eighth image as shown in Figure 12.
  • the eighth image of Figure 12 is CLOSED in step 48 in sequence with the fourth structuring elements shown in Figures 23-26, forming a ninth image as shown in Figure 13.
  • a UNION of the ninth image with the OPENINGS of the same set of four structuring elements shown in Figures 23-26 results in a tenth image free from most background noise as shown in Figure 14.
  • the tenth image of Figure 14 is reduced in step 52 by a factor of 2, forming an eleventh image as shown in Figure 15.
  • a bounding box fill is performed on the eleventh image of Figure 15 in step 54, forming the twelfth image of Figure 16.
  • the twelfth image in Figure 16 depicts a solid bounding box mask which is expanded as shown in step 56 to full scale, then logically ANDed with the first image (of Figure 5) from step 30, in AND gate 58, wherein the process of image markup detection is completed at step 60.
  • FIG. 35 Another preferred embodiment of a method for extraction of the contents of a closed circled region includes extracting handwritten marks at full resolution as shown in Figure 35 (which is the same result as Figure 33 of the previously described embodiment).
  • Figure 36 the bounding boxes of the connected components have been obtained.
  • each subimage is extracted and handled separately.
  • Figures 37-42 show results at intermediate stages of processing, where the subimages have been put together in their original location, forming a composite image.
  • each subimage has been edge-filled (in a first edge-fill) by doing a flood fill from the edges for each subimage. If the hand mark is closed, the fill will not penetrate the interior.
  • each subimage is inverted (in a first, edge-fill invert). If the hand mark contour is open, there will be little left (just the holes of text, for example). If the circle is closed, the contents appear as white on a black background. Note that the previous two steps can be used to decide whether a closed contour exists by testing for the number of ON pixels compared to the area of contour. If the ON pixels occupy a significant fraction of the area (for example, more than 30%), then the contour is closed. Alternatively, if after a small EROSION there are no ON pixels left, the contour is open.
  • each inverted subimage can be edge-filled (in a second edge-fill) as shown in Figure 39.
  • a solid subimage of ON pixels results for open contours, whereas the edge-fill just fills the exterior part of the closed contours.
  • Figure 40 shows each subimage inverted (in a second edge-fill invert) whereby the interior is extracted.
  • the two open contours have no interior, whereas the closed contour subimages yield the contents shown. Since the graphic subimage appeared as a closed contour, its non-zero contents also are displayed.
  • the basic operation of content extraction is edge- fill/invert, which is a topological operation extracting the interior ON pixels within a closed contour, noting that the shape of the various image components is not important.
  • Another topological method for extracting the contents of closed circled region includes: scanning a document image in step 250; filling the document images from the edges in step 252 (which fills the whole image except enclosed regions); bitwise inverting the image in step 254 so that only the enclosed regions are filled; filling from the edges in step 256; and bitwise inverting the image in step 258, resulting in an image with only hand drawn circled regions filled, the first edge-fill operation is provided to exclude machine made encircled regions.
  • a semi-topological method for extracting the contents of a closed circled region is shown in Fig. 45.
  • the method includes: scanning a document image in step 280; filling the document image from the edges in step 282; bitwise inverting the image in step 284; OPENING the image using a solid structuring element in step 286; CLOSING the image using at least one vertical structuring element in step 288; and logically ANDing the CLOSED image with the original image scanned in step 280, resulting in an image which includes only those regions which have been hand drawn as circles.
  • FIG. 41 shows the result after an opening. Selecting only the non-images; i.e., those whose contours are closed (as described in the first edge-fill invert above) the result, in Figure 41, is that there are now only two subimages. Then each subimage can be closed with a SE large enough to convert the background text to ON pixels, so that the resulting bitmap is a region of solid ON pixels. After closing using a SE that corresponds to 0.13 inch in the original image (the size can vary), the result in Fig. 42 is obtained in two ways:
  • the addBorderPixar and removeBorderPixar operations are a small implementation detail to prevent problems at the image boundaries in the CLOSING operation. Since there is always a border of white space, there are no image boundary issues.
  • the image hand mark detection device 80 of Fig. 4 includes, for example, a processor 84, monitor 86, read only memory 88, random access memory 90, files 92 and an output device 94.
  • Device 80 represents any number of automatic devices for altering a document image such as a computer system, a facsimile system or a photocopying system.
  • the user input device 80 represents any user input device such as a keyboard, optical scanner electronic scanner, photocopier or facsimile machine.
  • the processor 84 operates by way of instructions provided by ROM 88 and information provided by RAM 90 whereby access to files 92 can be obtained.
  • the results of the extraction of the hand marked document image segments can be displayed in monitor 86 or output to output device 94 (which represents any output device for outputting a document image such as a facsimile machine, photocopier, printer or CRT display).
  • the previously described apparatus and methods have shown how to identify and extract markup lines from a source image, using characteristic features of both the markup lines and the other print in the source image.
  • the current invention has used only morphological operations and threshold reductions wherein all morphological operations are translationally invariant and can be performed with both parallel and pipelined architectures, thus affording extremely fast and cost effective implementations.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Character Input (AREA)
  • Image Input (AREA)
  • Editing Of Facsimile Originals (AREA)
  • Image Analysis (AREA)
EP92310438A 1991-11-19 1992-11-16 Procédé et appareil de détection de marquage à main d'images Expired - Lifetime EP0543599B1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US794275 1991-11-19
US07/794,275 US5201011A (en) 1991-11-19 1991-11-19 Method and apparatus for image hand markup detection using morphological techniques

Publications (3)

Publication Number Publication Date
EP0543599A2 true EP0543599A2 (fr) 1993-05-26
EP0543599A3 EP0543599A3 (en) 1993-12-22
EP0543599B1 EP0543599B1 (fr) 2000-02-02

Family

ID=25162194

Family Applications (1)

Application Number Title Priority Date Filing Date
EP92310438A Expired - Lifetime EP0543599B1 (fr) 1991-11-19 1992-11-16 Procédé et appareil de détection de marquage à main d'images

Country Status (4)

Country Link
US (1) US5201011A (fr)
EP (1) EP0543599B1 (fr)
JP (1) JP3305772B2 (fr)
DE (1) DE69230635T2 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0780797A3 (fr) * 1995-12-21 1997-09-17 Adobe Systems Inc Procédé et système pour la désignation d'objets
GB2362056A (en) * 2001-02-09 2001-11-07 Hewlett Packard Co Detecting and saving annotations in a scanned document
US7042594B1 (en) 2000-03-07 2006-05-09 Hewlett-Packard Development Company, L.P. System and method for saving handwriting as an annotation in a scanned document

Families Citing this family (54)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH05151254A (ja) * 1991-11-27 1993-06-18 Hitachi Ltd 文書処理方法およびシステム
JP3422541B2 (ja) * 1992-12-17 2003-06-30 ゼロックス・コーポレーション キーワードのモデル化方法及び非キーワードｈｍｍの提供方法
US5438630A (en) * 1992-12-17 1995-08-01 Xerox Corporation Word spotting in bitmap images using word bounding boxes and hidden Markov models
JP3272842B2 (ja) * 1992-12-17 2002-04-08 ゼロックス・コーポレーション プロセッサベースの判定方法
JPH0756956A (ja) * 1993-07-22 1995-03-03 Xerox Corp 人間の作成したイメージに基づくデータアクセス方法
US5519618A (en) * 1993-08-02 1996-05-21 Massachusetts Institute Of Technology Airport surface safety logic
US5374932A (en) * 1993-08-02 1994-12-20 Massachusetts Institute Of Technology Airport surface surveillance system
US6415048B1 (en) 1993-10-12 2002-07-02 Schneider Medical Technologies, Inc. Compositional analysis system
EP0701224B1 (fr) * 1994-09-09 2000-12-06 Xerox Corporation Méthode d'interprétation de commandes de l'interface utilisateur tracées à la main et schématiques
US5555556A (en) * 1994-09-30 1996-09-10 Xerox Corporation Method and apparatus for document segmentation by background analysis
US5729741A (en) * 1995-04-10 1998-03-17 Golden Enterprises, Inc. System for storage and retrieval of diverse types of information obtained from different media sources which includes video, audio, and text transcriptions
US5966136A (en) * 1995-04-12 1999-10-12 Hewlett-Packard Co. Efficient method for clipping numerous objects against an arbitrary clipping path
EP0738987B1 (fr) * 1995-04-21 2004-11-10 Xerox Corporation Traitement des formulaires compréhensibles par une machine
US5787208A (en) * 1995-06-07 1998-07-28 Neopath, Inc. Image enhancement method and apparatus
WO1997009690A1 (fr) * 1995-09-05 1997-03-13 Northrop Grumman Corporation Criblage dimensionnel de donnees et connectivite floue pour analyse d'image d'irm
US5937077A (en) * 1996-04-25 1999-08-10 General Monitors, Incorporated Imaging flame detection system
US5692073A (en) * 1996-05-03 1997-11-25 Xerox Corporation Formless forms and paper web using a reference-based mark extraction technique
US6192160B1 (en) * 1996-09-19 2001-02-20 Hyundai Microelectronics Co., Ltd. Hardware architectures for image dilation and erosion operations
US6275304B1 (en) * 1998-12-22 2001-08-14 Xerox Corporation Automated enhancement of print quality based on feature size, shape, orientation, and color
US6867875B1 (en) 1999-12-06 2005-03-15 Matsushita Electric Industrial Co., Ltd. Method and apparatus for simplifying fax transmissions using user-circled region detection
US6597808B1 (en) * 1999-12-06 2003-07-22 Matsushita Electric Industrial Co., Ltd. User drawn circled region extraction from scanned documents
US6766069B1 (en) * 1999-12-21 2004-07-20 Xerox Corporation Text selection from images of documents using auto-completion
US7123768B2 (en) * 2000-04-26 2006-10-17 Minolta Co., Ltd. Apparatus and method for detecting a pattern
JP3867512B2 (ja) * 2000-06-29 2007-01-10 富士ゼロックス株式会社 画像処理装置および画像処理方法、並びにプログラム
JP2002150306A (ja) * 2000-09-04 2002-05-24 Minolta Co Ltd 画像処理装置、画像処理方法、画像処理プログラムおよび画像処理プログラムを記録したコンピュータ読取可能な記録媒体
DE10215548B4 (de) * 2002-04-09 2006-05-11 Eltromat Polygraph Gmbh Verfahren und Vorrichtung zur Erfassung von Abtastpositionen in Druckbildern
DE60302191T2 (de) * 2002-08-27 2006-07-27 Oce Print Logic Technologies S.A. Ermittlung der Schräglage von Dokumentenbildern
US7283685B2 (en) * 2003-09-23 2007-10-16 Microtek International Inc. Device that appends a recognition point for image joining to the extracted image and a recognition element thereof
GB0326374D0 (en) * 2003-11-12 2003-12-17 British Telecomm Object detection in images
US7834819B2 (en) 2004-04-01 2010-11-16 Polyvision Corporation Virtual flip chart method and apparatus
US7948448B2 (en) 2004-04-01 2011-05-24 Polyvision Corporation Portable presentation system and methods for use therewith
US7567714B2 (en) * 2004-07-07 2009-07-28 The United States Of America As Represented By The Secretary Of The Navy System, method and apparatus for clustering features
KR100630732B1 (ko) * 2005-01-11 2006-10-02 삼성전자주식회사 블록 단위 병렬 세선화 방법 및 장치
US20070115500A1 (en) * 2005-11-21 2007-05-24 Xerox Corporation Method for operating communication device
JP4933993B2 (ja) * 2006-09-19 2012-05-16 株式会社リコー 画像管理方法及び画像処理装置
KR20080055119A (ko) * 2006-12-14 2008-06-19 삼성전자주식회사 화상형성장치 및 그 제어방법
JP4332858B2 (ja) * 2007-03-02 2009-09-16 ブラザー工業株式会社 画像処理装置および印刷装置、画像処理方法
US8009928B1 (en) 2008-01-23 2011-08-30 A9.Com, Inc. Method and system for detecting and recognizing text in images
US20090251468A1 (en) * 2008-04-03 2009-10-08 Peled Nachshon Animating of an input-image to create personal worlds
JP4966912B2 (ja) * 2008-04-21 2012-07-04 大日本スクリーン製造株式会社 線画処理装置、線画処理方法およびプログラム
US8379101B2 (en) 2009-05-29 2013-02-19 Microsoft Corporation Environment and/or target segmentation
US8452086B2 (en) * 2009-07-10 2013-05-28 Palo Alto Research Center Incorporated System and user interface for machine-assisted human labeling of pixels in an image
US8649600B2 (en) 2009-07-10 2014-02-11 Palo Alto Research Center Incorporated System and method for segmenting text lines in documents
US8442319B2 (en) * 2009-07-10 2013-05-14 Palo Alto Research Center Incorporated System and method for classifying connected groups of foreground pixels in scanned document images according to the type of marking
US9218523B2 (en) * 2009-09-04 2015-12-22 Siemens Industry, Inc. System and method for determining authorship of a document
US9245043B2 (en) * 2009-12-23 2016-01-26 Fuji Xerox Co., Ltd. Embedded media markers and systems and methods for generating and using them
US8682075B2 (en) * 2010-12-28 2014-03-25 Hewlett-Packard Development Company, L.P. Removing character from text in non-image form where location of character in image of text falls outside of valid content boundary
JP6107270B2 (ja) * 2013-03-19 2017-04-05 ブラザー工業株式会社 画像処理装置およびコンピュータプログラム
JP6540280B2 (ja) 2015-06-30 2019-07-10 ブラザー工業株式会社 画像処理装置およびコンピュータプログラム
US10264213B1 (en) 2016-12-15 2019-04-16 Steelcase Inc. Content amplification system and method
US10049097B1 (en) * 2017-01-27 2018-08-14 Xerox Corporation Systems and methods for creating multi-layered optical character recognition (OCR) documents
CN111127534A (zh) * 2019-11-05 2020-05-08 深圳市三宝创新智能有限公司 一种障碍物的检测方法
CN111860479B (zh) * 2020-06-16 2024-03-26 北京百度网讯科技有限公司 光学字符识别方法、装置、电子设备及存储介质
US11930153B2 (en) 2021-01-08 2024-03-12 Hewlett-Packard Development Company, L.P. Feature extractions to optimize scanned images

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0431962A2 (fr) * 1989-12-08 1991-06-12 Xerox Corporation Méthode et appareil d'identification de l'inclinaison de documents
US5048109A (en) * 1989-12-08 1991-09-10 Xerox Corporation Detection of highlighted regions
US5048099A (en) * 1990-05-21 1991-09-10 Eastman Kodak Company Polygon-based method for automatic extraction of selected text in a digitized document
EP0490687A2 (fr) * 1990-12-13 1992-06-17 Xerox Corporation Procédé et appareil de traitement d'image

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4720750A (en) * 1986-02-28 1988-01-19 Kabushiki Kaisha Toshiba Image forming apparatus
JPS63158678A (ja) * 1986-12-23 1988-07-01 Sharp Corp 単語間スペ−ス検出方法
JPH0738681B2 (ja) * 1987-03-20 1995-04-26 富士ゼロックス株式会社 領域認識装置
JPH07101915B2 (ja) * 1987-09-21 1995-11-01 コニカ株式会社 画像処理装置
US4908716A (en) * 1987-12-08 1990-03-13 Ricoh Company, Ltd. Image processing apparatus
JP2683008B2 (ja) * 1988-02-16 1997-11-26 株式会社リコー 矩形図形内部塗りつぶし方法
JP2828645B2 (ja) * 1989-01-27 1998-11-25 株式会社リコー マーク領域判定装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0431962A2 (fr) * 1989-12-08 1991-06-12 Xerox Corporation Méthode et appareil d'identification de l'inclinaison de documents
US5048109A (en) * 1989-12-08 1991-09-10 Xerox Corporation Detection of highlighted regions
US5048099A (en) * 1990-05-21 1991-09-10 Eastman Kodak Company Polygon-based method for automatic extraction of selected text in a digitized document
EP0490687A2 (fr) * 1990-12-13 1992-06-17 Xerox Corporation Procédé et appareil de traitement d'image

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
C.R. GIARDINA & E.R. DOUGHERTY 'Morphological methods in image and signal processing' 1988 , PRENTICE HALL, INC. , ENGLEWOOD CLIFFS, US * page 111 - page 125 * * Ch. 4 Topological processing * *
OPTICAL ENGINEERING vol. 26, no. 7 , July 1987 , BELLINGHAM US pages 623 - 632 P. MARAGOS 'Tutorial on advances in morphological image processing and analysis' *
SYSTEMS & COMPUTERS IN JAPAN vol. 17, no. 3 , 1986 , NEW YORK US pages 35 - 46 Y. SUENAGA 'Some Techniques for Document and Image Preparation' *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0780797A3 (fr) * 1995-12-21 1997-09-17 Adobe Systems Inc Procédé et système pour la désignation d'objets
US5930813A (en) * 1995-12-21 1999-07-27 Adobe Systems Incorporated Method and system for designating objects
US7042594B1 (en) 2000-03-07 2006-05-09 Hewlett-Packard Development Company, L.P. System and method for saving handwriting as an annotation in a scanned document
GB2362056A (en) * 2001-02-09 2001-11-07 Hewlett Packard Co Detecting and saving annotations in a scanned document
GB2362056B (en) * 2001-02-09 2004-05-26 Hewlett Packard Co System and method for saving handwriting as an annotation in a scanned document

Also Published As

Publication number Publication date
EP0543599B1 (fr) 2000-02-02
JPH05250514A (ja) 1993-09-28
DE69230635D1 (de) 2000-03-09
DE69230635T2 (de) 2000-07-27
JP3305772B2 (ja) 2002-07-24
US5201011A (en) 1993-04-06
EP0543599A3 (en) 1993-12-22

Similar Documents

Publication Publication Date Title
EP0543599B1 (fr) Procédé et appareil de détection de marquage à main d'images
KR100251600B1 (ko) 이미지 처리 방법 및 디지탈 컴퓨터
US5570435A (en) Segmentation of text styles
JP3086702B2 (ja) テキスト又は線図形を識別する方法及びデジタル処理システム
US5390259A (en) Methods and apparatus for selecting semantically significant images in a document image without decoding image content
CA2077565C (fr) Methode et dispositif de modification automatique de parties a contenu semantique dans un document sans decodage d'image
JP3051182B2 (ja) ハイライト領域検出方法
US5491760A (en) Method and apparatus for summarizing a document without document image decoding
US5828771A (en) Method and article of manufacture for determining whether a scanned image is an original image or fax image
US6411733B1 (en) Method and apparatus for separating document image object types
Tan et al. Text extraction using pyramid
JP3345224B2 (ja) パターン抽出装置、パターン再認識用テーブル作成装置及びパターン認識装置
JP3913985B2 (ja) 文書画像中の基本成分に基づく文字列抽出装置および方法
US5455871A (en) Detecting function words without converting a scanned document to character codes
Lakshmi et al. An optical character recognition system for printed Telugu text
StevensÝ et al. Automatic processing of document annotations
Fateman How to find mathematics on a scanned page
JPH07105312A (ja) 光学式文字読取装置における文字イメージのごみ除去方法及び装置
JP2008108114A (ja) 文書処理装置および文書処理方法
Baig et al. Automatic segmentation and reconstruction of historical manuscripts in gradient domain
Dhandra et al. Word-wise script identification from bilingual documents based on morphological reconstruction
Srinivas et al. An overview of OCR research in Indian scripts
US20050238235A1 (en) Run length based connected components and contour following for enhancing the performance of circled region extraction algorithm
JP3476595B2 (ja) 画像領域分割方法、および画像２値化方法
JP3067474B2 (ja) 画像処理装置

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): DE FR GB

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): DE FR GB

17P Request for examination filed

Effective date: 19940601

17Q First examination report despatched

Effective date: 19961219

GRAG Despatch of communication of intention to grant

Free format text: ORIGINAL CODE: EPIDOS AGRA

GRAG Despatch of communication of intention to grant

Free format text: ORIGINAL CODE: EPIDOS AGRA

GRAH Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOS IGRA

GRAH Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOS IGRA

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): DE FR GB

REF Corresponds to:

Ref document number: 69230635

Country of ref document: DE

Date of ref document: 20000309

ET Fr: translation filed
PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed
REG Reference to a national code

Ref country code: GB

Ref legal event code: IF02

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20041111

Year of fee payment: 13

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20050825

Year of fee payment: 14

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20060601

REG Reference to a national code

Ref country code: FR

Ref legal event code: ST

Effective date: 20070731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20061130

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20101110

Year of fee payment: 19

REG Reference to a national code

Ref country code: GB

Ref legal event code: PE20

Expiry date: 20121115

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF EXPIRATION OF PROTECTION

Effective date: 20121115