WO2001001231A1 - Designation system and method for interactive computer-networked study aid - Google Patents

Abstract

A method for designating questions and answers in a computer-networked self-study system also lends itself to discriminating certain portions of a larger work for use in conjunction with other systems requiring text discrimination in a selectable manner. When a student wants to add an additional question and answer couple to a self-study system, both the question and answer are highlighted: the question in one color and the answer in another. The text is then scanned in to a computer in a manner that preserves the color information. The highlighted text is then subject to extraction and decolorization with the distinctions present in the text afforded by the different colors preserved. The text may then be subject to optical character recognition (OCR) or the like. The question portion is then used to supply a question to a student. Any answer given by the student during a study session is compared to the answer previously indicated by highlighting. The method herein is also effective with regards to testing (indicating the correct answer) and forms (for information gathering). Additionally, other subject matter other than text, such as graphics, may be made subject to the present system. Additionally, any detectable marking means may be used that conforms to the operation of the present invention.

Description

DESIGNATION SYSTEM AND METHOD FOR INTERACTIVE COMPUTER-NETWORKED STUDY AID

TECHNICAL FIELD

This mvention relates to mteractive computer systems, and more particularly to a novel process to isolate a group of pixels, naming a spatial segment of a scanned-in document, or otherwise, with respect to the entire input page to allow segregation and special recognition mdicated text, drawmg, or graphic In a preferred embodiment, text is highlighted in different colors for questions and answers The highlighted text may then be used for a self- study system, image processmg, or information processing management system

BACKGROUND ART

Text-designation systems are known in the art Listed below are patents generally relevant to the present invention

Patent Number Inventor Date of Issue 6,028,601 Machiraju et al February 22, 2000 6,017,219 Adams, Jr et al January 25, 2000 5,982,370 Kamper November 9, 1999 5,978,799 Hirsch November 2, 1999 5,893,914 Clapp April 13, 1999 5,893,717 Kirsch et al April 13, 1999 5,772,446 Rosen June 30, 1998 5,649.024 Goldsmith July 15, 1997 5,610,665 Berman March 1 1 , 1997 5,519,608 Kupiec May 21 , 1996 5,451,163 Black September 19, 1995 5,303,042 Lewis et al April 12, 1994 5,257,185 Farley et al October 26, 1993 5,035,625 Munson et al July 30, 1991 5,002,491 Abrahamson et al March 26, 1991

A more in-depth description of each of the more pertinent patents follows

Machiraju et al., U.S. Pat. No. 6,028,601

This reference is directed to a FAQ link creation between user's questions and answers As a user types a question in query field 210, the system executes an algorithm to remove all stopwords m the question and to stem the remaining words The user may then select a retrieved question by pomtmg a mouse-controlled pointer 260 to the retrieved question and double-chckmg the mouse A retrieved question may also be selected by using a screen cursor

265 to highlight the question By selecting a retrieved question, the linked sections from the document collection 220, as shown m Figure 4, are automatically opened for viewing

Figure 8 is a screen shot 350 before creation of a link between the question 300 and the document texts in ranked document list 310 of Figure 7 The user first highlights a text section in the document 340 to create a link between the highlighted text section and the question 300 The user then creates the link between the question 300 and the highlighted text section by clicking the "add link" selection 355 The question 300 can be linked to multiple text sections in one or more documents

Kirsch et al., U.S. Pat. No. 5,893,717

This reference is directed to a computerized method and system for teachings prose, document and quantitative literacy The example shown when "length" is selected from the menu is in Figure 28 and is accompanied by the following text can this cleanmg pad be used with rubber kitchenware⁹ Students must click on "yes" or "no" Clicking ' more" highlights the answer and puts up feedback a list can be hard to use if you have to look through lots of items to find an answer "Rubber kitchenware" is highlighted in the question and m the list

Kupiec, U.S. Pat. No. 5,519,608

This reference is directed to a method for extracting from a text corpus answers to questions stated in natural language by using linguistic analysis and hypothesis generation The system accepts a natural-language mput string such as a user-supplied question and a set of relevant documents that are assumed to contain the answer to the question In response, it generates answer hypotheses and finds these hypotheses withm the documents The mvention highlights the answer hypothesis in the retrieved document for the user's convenience In step 287, the ranked hypotheses are organized into results suitable for output In one embodiment in which results are to be presented to the user, the highest- ranked answer hypothesis is selected for presentation This hypothesis is highlighted in the contexts in which it appears in primary and secondary documents, for example, by displaying the document titles and the match sentences that confirm the linguistic relations implied by the user's question

Black, U.S. Pat. No. 5,451,163

This reference is directed to a method of teaching reading including displaying one or more visible symbols in a transparent medium between student and teacher In the final portion of the lesson, questions are produced on the video screen accompanied by the visual pronunciation of the question by the leader The answer to the question is introduced and then highlighted character-by-character, thereby helpmg the student to produce the sound of each character In an alternative embodiment, the questions may be presented m video only, or in audio only

Adams, Jr. et al., U.S. Pat. No. 6,017,219

This reference is directed to a system and method for mteractive reading and language instruction The positional pacer 17, with mput from the local text position management data bases 22, may be implemented to contmuously prompt the student along the text, identifymg the word that is bemg read by the computer mstructor or is to be pronounced by the student In addition, the executive program may highlight or color differentiate the text to be read by the computer and that to be read by the student DISCLOSURE OF INVENTION

The present application is related to PCT international application serial number PCT/US00/12686 filed May 8 2000 for an Interactive Computer Networked Study Aid and Guide, which application is incorporated herem by this reference thereto In that application, reference is made to a highlighting system where, in a question and answer marker function, the user may highlight part of a paragraph or sentence with a pomtmg device, handheld scanner and storage, or otherwise to indicate text that may form a study question The highlighted question may then be cut and/or may be shown m a different color Such colors may change from question to question Upon designating the question, the user then highlights the part of the paragraph or sentence that forms the answer to the previously-highlighted question and then designates the highlighted text as by cutting it from soft text, hard copy, or otherwise The highlighted question portion is then placed in a question portion of a template and the text selected as the answer may be omitted from the question-designated text An underline may be inserted m the place of the answer text that has been cut out to leave a blank to be filled in by the student-supplied answer The answer is placed in the answer portion of the template to provide the correct answer for the student

In the present mvention, the designation system inspects a graphic file for blocks of a specific color or optical image and copies the blocks from the file and pastes them in the desired positions in the template, document, or other computer file Each color has a different designation final location For example, questions might be marked in yellow while answers might be marked m blue by marking the text in a hard copy and scannmg it or by highlighting electronic text Questions and answers are then generated for use in the BrainX system By so indicating the questions and answers, the student may generate study materials used m the BrainX system

Methods of marking are not limited to the embodiments disclosed herem or the visible spectrum as a whole, but any system of marking that may be used to distinguish optically or otherwise selected text The present system includes the use of other marking methods including disappearing or erasable ink or the like that do not leave permanent marks on a document As used herein, the term 'text" generally includes any written communication including printed or handwritten text as well as drawings and graphics

The BrainX™ scanning process of the present mvention ("BSP") includes a set of algorithms and processing steps that take as input a demarcated region of text/print/drawing/picture of any type The output from the process includes all of the pixels in the boundmg rectangle of space (on the original document) This output is generally free from noise, in the form of coloration or random pixels, and descπbes the originally marked section of text This bounded region has contextual significance to a final programmatic post-process, which may be OCR, other forms of image or text processmg, or the like Depending upon unique mput cues, that become part of the discriminatory BrainX™ scannmg process, the output may then be used mtelligently, as mput to another application or program, for display, or further processing The cues that trigger the discrimmation of area, for scannmg, are m the form of highlighted coloration of the area of interest withm the larger document to be used as mput (Figure 2) By usmg colored-highlighted text, or area, as mput of interest, the BSP can present for subsequent display or post-processing, an arbitrary subset of words or pictures, These bemg derived directly, and extracted from, a larger set of prmted material scanned and input to a data processmg device by one of several different means

By employmg highlighting as the mput cue, or hmt, the BrainX process allows the operator, or editor, to choose the subset of pπnted material of interest, by simply marking the target areas with a highlighting marker pen, manually The operator, therefore, needs not be expert in scannmg, computer processmg, or any other technical process normally involved when such material is to become data mput for a modern computational device We also claim another benefit, which accrues by the ability withtn this process to discriminate between differently colored "cueing" markers, thereby creating a plurality of contexts which may become individual and separate subsets of mput, each havmg different contextual meanmg, for further computational processmg In this way, material of contrasted meanmg or context, can be automatically sorted and pre-processed The various sets of analogous material can then be presented, for onward manipulation, havmg already been automatically classified

BRIEF DESCRIPTION OF DRAWINGS

Figure 1 is a schematic representation of partial text extraction from a larger textual work

Figure 2 a representation of highlighted text used as a keymg mechanism for the present invention

Figure 3 is an example of a pixel map

Figure 4 is an schematic indication of scanner technology known m the art

Figure 5 is a representative progression showing color space reduction as used in the present mvention

Figure 6 shows a convolution matrix as used m the present mvention

Figure 7 shows a template matrix as used m the present mvention

Figure 8 is a spatial representation of colors represented as points in space

Figure 9 is an exemplary color histogram for an arbitrary given area

Figure 10 shows a graphic mdicatmg the border tracing process as used in the present invention

Figure 11 is an exemplary test prior to answer highlighting as may be provided by the present invention

Figure 12 is the exemplary test of Figure 1 1 with answers highlighted

Figure 13 shows highlighted text as may be used to designate text in the present invention

Figure 14 shows a form that may be highlighted for use in conjunction with the present invention

MODE(S) FOR CARRYING OUT THE INVENTION

The detailed description set forth below in connection with the appended drawings is intended as a description of presently-preferred embodiments of the mvention and is not intended to represent the only forms which the present invention may be constructed and/or utilized The description sets forth the functions and the sequence of steps for constructing and operating the mvention m connection with the illustrated embodiments However, it is to be understood that the same or equivalent functions and sequences may be accomplished by different embodiments that are also intended to be encompassed withm the spirit and scope of the mvention

The BrainX™ BSP is used primarily as an intelligent agent process, to enhance, or facilitate, the rendermg of chosen textual or artistic material, m the context of its marked-up coloration on the prmted page, scanned page, soft (electronic) text, or otherwise In other words, an operator may use colored marking devices, the form of pencil or marker pen, and by highlightmg text or drawmg upon a prmted page (which is then scanned by the BSP), can extract that material from its greater context, mto discrete segments These discrete segments of text or drawn material are then further processed by the BSP, to remove the highlight-coloration, and any other noise or artifacts that may be part of the scanned mput process

By usmg coloration-highlighting as the key, or hint, the BSP can effectively choose and extract, m an intelligent (but operator friendly) way, any and all context-independent material that is originally part of a greater homogeneous document, or piece of prmted material (Figure 1)

The BSP is used to enhance and enable the manual extraction of finite and discrete elements, from a greater whole, by facilitating the automatic demarcation of those elements with simple tools and lack of operator complexity As an example, it is envisioned that this process may be used to markup a document for testing or editmg purposes, by usmg simple contextual keys, the highlightmg of the areas mvolved, with simple colored highlightmg markers or pens (Figure 2)

The theories mvolved to accomplish the task by using the BSP, are in the domam of signal and image processmg, as well as OCR (Optical Character Recognition), and temporal/spatial color theory Additionally, the mechanics and techniques of electronic document scanning are touched upon, for completeness

The most popular way of bringing images mto a computer, namely document images, is with a scanner Modern scanners come m many forms, from handheld to 1000 page automated feeding machmes Some scanners can store information for upload to a computmg device Flatbed scanners are the most familiar and ubiquitous A copier is an example of a flatbed scanner Flatbed scanners can be equipped with document feeders to elunmate slow manual paper handling from the scannmg process

The active spectrum of such scanners may include frequency detection of light outside the visible spectrum Chemical or other scanners may also be put to good use in the achievement of the present invention Particularly, documents that have been marked with emissive or other indicating materials may be susceptible to chemical scannmg

In a flatbed scanner, the document to be scanned is typically held still while the scanner's CCD is drawn across it Alternatively, the scanner's CCD can be held still while the document is pulled through the scanner, which is what is done in portable scanners

While the BSP process relies upon data that is the ultimate output product of the raw scannmg process, BSP is scanner device independent and does not rely directly upon either the device, or any other hardware

There are a number of ways to attach a scanner to a computer SCSI, parallel, and video interfaces are used for the physical-electronic connection between a scanner and computer The BSP process is interface- independent and does not rely directly upon either the computational device, or its interface hardware

The software used to interface a scanner with the computational device is typically an ISIS or TWAIN driver These are software mterface elements that allow a specific scanner to communicate with it's controlling computational and storage device The BSP process is dπver-mdependent and does not rely upon either the mput device, or its driver software

Computer displays work by a process called raster scannmg An electron beam sweeps across the screen, building up the picture one scanline at a time (Figure 3)

Documents are scanned mto a computer via a similar rasterization process As the scannmg head passes over the paper, or a piece of paper feeds through the scanner, a Charged Coupled Device (CCD), or similar hardware, reads the document one line at a tune Since it is known that mk absorbs light and blank paper reflects it, the CCD acts as the opposite of the phosphor, sensmg dark and light spots on the paper The scanner's embedded computational device generates a series of 1 s and 0s corresponding to dark and light spots on the oπgmal document

Many scanners will recreate full color images The BSP relies upon this feature Color scanners scan for the primary colors, red, blue, and green A color filter is sometimes used to separate the colors the scanner

The target image is a set of pomts m a plane, each with its own luminance and color The BSP can differentiate between bmary images (havmg only two distmct lummance values), grey-value (monochrome) images and color images The BSP uses image processmg to manipulate the mput images m various ways, in order to

• Code the image, for reduction of required storage space and to facilitate processmg,

• Reconstruct the image, reducmg distortion and / or noise,

• Enhance the image, as pre-processmg for further interpretation,

• Extract information from the image

The color set representation is defined as follows

Assume that each of the possible colors in the target color image may be described by a triple (r, g, b) from the 3-D rgb color space Without loss of generality, assume that

is one of many possible transformations that are not necessarily lmear between RGB and another color space denoted XYZ For each (r,g,b) the triple (x,y,z) represents the transformed color such that,

ι , y, ι) = T { r, g, b)

where

is a conversion from RGB to XYZ

Let

9; ⁽ . *) be quantizer function that maps each value of to one of

Then let

represent the quantized color pomt such that,

y = 9 ι (y. *

and x = «j, t , n)

Let the vector

*, *

Represent the point (*, y, *)

It follows that

? —

is one of

I m n possible vectors m the quantized color space

XYZ

The BSP color set algorithm provides simultaneously a technique for extraction and for quick and efficient indexing The construction of the color set mdex is similar conceptually to file inversion (fully indexed by color set, location — and vice versa), but where the locations of the occurrences of color withm an unage are kept in a list which is ordered by color This works well for unage area colors, because the full set of visible colors can be represented without visible distortion usmg a fixed finite set of colors Using a file inversion type of query, first the selected colors are identified Then the BSP procedure consults the color list and reports on the occurrences of the query colors across the unages m the image database When the color list points not to individual pixels but to regions, and the indexing is based upon color sets contammg possibly many colors, the color set approach enables the procedure to have great power in the indexing of color images

The success of the BSP color set procedure relies on a reduction of the dimensionality of the color feature space and the ability to satisfactorily localize color mformation spatially withm unages In short, as illustrated in Figure 5, this is accomplished by the following means reduction of the full gamut of colors to a set of manageable size (-100 carefully selected colors) This involves selection of

And

?, (», v)

A primary procedure objective is that unacceptably dissimilar colors are not mapped mto the same bins Deference also allows higher tolerance for dissimilarity in color lightness and color saturation while reserving the most fine quantization for hue A BSP colorizing' algorithm characterizes the color areas usmg the reduced palette This ensures that the most dommant colors and regions are emphasized while msignificant color mformation is dropped After this processmg a conditional search over the sets of colors remammg in the image reveals the spatially localized color regions The regions that are sufficiently represented by a color set are mapped mto the database mdex to be retrieved through selection of the color set

The BSP input unage can be mathematically denoted as a real function of two real values Let S be the set of image points and let x and y be coordinates m the unage, then the unage can be described by the function f(x, y), where

/[ , y) = 0, Vz, t, ^ and

0 < /(z, £_/) < M, V.r, s_/ £ tf

(with M being the maximum luminance in the image). Since f(x, y) is a absolutely integratable function, the two- dimensional Fourier transform of the function exists if its number of discontinuities is finite. This Fourier transform is defined as:

f(x, y) is a representation in the space domain and F(u, v) a representation in t e frequency domain.

In order to process the input BSP image using a computer, it first is digitized. This process is a combination of quantizing (conversion of the luminance-scale from continuous to discrete) and sampling (conversion of the x-y plane from continuous to discrete). The two-dimensional discrete Fourier transform (DFT) of the image is:

Convolution is a mathematical operator (denoted by "*") on two functions, defined as the sum of the products of the function values, with one of the functions mirrored in the origin. The convolution value is defined for every relative position of the two target BSP functions. Note should be taken that for common software processes, the asterisk ("*") is often used to indicate multiplication. However, for digital signal processing, the asterisk ("*") is used to indicate convolution. Context will indicate herein which meaning is to be used for the asterisk ("*") symbol.

Let

and

9{* Va) be the two target BSP functions Then their convolution will be defined as

ac co

α =-ao b=-ao

A property of the convolution process is that the convolution in the space domain is equivalent to multiplying m the frequency domain and vice versa,

let N be the number of pomts in the BSP Fourier transform, then

The correlation of the two BSP functions is then defined as the sum of the products of the function values, for every relative position of the two functions The correlation of the BSP function and itself is further referred to as the BSP autocorrelation

Let

and

9{^χ ₃ y₃) be the two BSP input functions Then correlation is then defined as

ao ao ^φ fa y) = 1 /( + <*> & + &) α=-ao _B=_a

The BSP uses filtering m the context of the mappmg of a local area around a pixel (mcludmg that pixel) onto the same pixel The local area, in this context, is defined as bemg larger than a smgle pixel, but smaller than the entire image BSP filtering is thus part of the larger class of local BSP algorithmic operations

Filters can be subdivided in lmear and non-linear filters Linear filter operations can be calculated by convolving the image with the filter (or, equivalently, by a multiplication m the frequency domam)

Usually, a discrete unage filter f(x, y) is represented by a wmdow contammg the function values for positions relative to the center of that wmdow The center value then is f(0,0) An example is the local average or uniform filter is shown m Figure 6 The BSP uses filters for the following objectives

• Smoothing or sharpening of the mput unages,

• Findmg image edges (gradients), and

• Input image restoration (filtering to reverse the effects of a linear distortion, e g focus or motion blur) Template matchmg is used as a BSP technique to isolate features in the mput image These features might be single pixels, lmes. edges or complete objects

An example is a template to find isolated pixels (1 e , greyvalue 1") m an image (with background 0") as in Figure 7

The BSP algorithms use template matchmg various matchmg criteria This is usually a threshold, applied to the image resultmg from the convolution between an unage and a template The threshold insures that mput features are detected only if they result m a large response in the convolved unage

In the example above, the threshold would be 8, for if only the central pixel is replaced with 8, there are no neighbormg pixels Values in the range [0 7] mdicate there is a central pixel, surrounded by one or more neighbors Values below 0 indicate there is no central pixel

Withm the primary BSP mput discriminator procedure, a color matchmg algorithm is employed The idea of the algorithm is that the color in the current color table that is closest to the requested color will be selected Closest is defined m terms of the normal distance metπc on the RGB cube If the closest color is farther away than the percentage error allowed, times the length of the diagonal of the RGB color cube, then an error number is issued together with a warning The closest color is still assigned for the requested color

To actually perform the work of color matching, the BSP uses the least squares method The RGB coordinates represent distances on three axes, reducing the problem to findmg the shortest distance between two pomts in three- dimensional space, as shown in Figure 8

The distance between two pomts is determined by taking the square root of the sum of the squares of the distances in three directions, corresponding to the three axes in Cartesian coordmates The formula is shown

*(x2-xl)2 + (y2-yl)2 + (z2-zl)2

Similarly, the distance between two points m color space is

*(R2-R1)2 + (G2-G1)2 + (B2-B1)2

The BSP color matchmg algorithm finds the distance between the mput background-(bounded area) color and all the other colors, then chooses the mdex with the shortest RGB distance The algorithm loops from

l = 0 to K (255*3)

groups of three

This gives an index for traversing the mput palette of possible cuemg colors The input palette array stores the red, green, and blue components of the color, so the algorithm mcrements by three for each color // find closest blue color usmg least squares method

distance = (63 - ιnput_palette[ι+2]) * (63 - ιnput_palette[ι+2]) + (21 - ιnput_palette[ι+l]) * (21 - ιnput_palette[ι+l]) + (21 - ιnput_palette[ι]) * (21 - ιnput_palette[ι]),

For efficiency, the square root is not taken The BSP algorithm is not concerned with the actual distance between pomts, just the relative distances, which will be consistent m the squares

The BSP algorithm would then do a compare for the distance to the previous value for blue (which is initialized to a high value)

if (distance < blue value)

{ blue = ι/3, blue value = distance,

If the distance is the lowest, the match is made This continues m a loop until all 255 colors are exammed It is possible to have no blue value m the mput palette When this happens, the algorithm will choose a shade of gray or cyan, or whatever color is closest to blue

Another portion of the BSP algorithm for characterizing unage content uses color histograms The color histogram for an image is constructed by countmg the number of pixels of each color The BSP algorithm follows this progression

• selection of a color space

• quantization of the color space

• computation of histograms

• derivation of the histogram distance function

• identification of indexmg shortcuts

The BSP color histogram is computed by discretizing the colors withm the unage and countmg the number of pixels of each color Smce the number of colors is finite, a smgle variable histogram is used

The BSP uses the QBIC system, which requires manual segmentation of images In QBIC, the color histogram is computed as attributes of the regions that have been outlmed manually This reduces the potential contribution of background and other irrelevant colors but requires human mvolvement m identification of the data (A colored marker) as shown in Figure 9

The BSP algorithm for removmg the primary highlight coloration is very simple

For every pixel = (target color mdex to remove) Logical AND with (background color)

This yields all pixel values withm the target area as either a mix of the (character/foreground color + highlight) or (background color)

From that pomt on, logical OR all pixels with Foreground color to normalize, then apply noise removal (next step)

If the scanned unage is noisy, BSP will reduce this noise to signal ratio by usmg threshold noise reduction (TNR) By usmg a progressive search mechanism, the algorithm isolates pixels patterns with a Noise Removal Granulaπty of 2x2 The means isolate pixels-dots equal or less than 2x2 are removed This algorithm can progressively remove more or less noise by setting the thresholds, 5x5 means isolate pixel-dots equal or less than 5x5 are removed Therefore when BSP progresses to usmg 5x5, most 4x4, 3x3 and 2x2 isolated pixel-dots are removed Usually, it will be sufficient to default the algorithm to one pass at a reasonably large pixel-dot starting point

After initial noise removal takes place, and additional pass through the BSP median filtering algorithm is done The algoπthm is non-linear, and uses a 3x3-search matrix to derive the median m of the set of values at any pomt in the image, where half the values m the set are less than m and half are greater

The outcome of this median filtering, is that pixels in the BSP input image with outlying values are forced to become more like their neighbors, but at the same time edges are preserved

The BSP algorithm starts at the left hand side of each area (of the histogram-derived segmented image) and locate the first black pixel This may be the bottom half of a character of text in the line above the target, the page number of the page, some noise from background areas of the mput media, or similar

Since the algorithm can derive estimates of the character height obtained from the greater bounding rectangle, the BSP can make an estimate of where the bottom of the character should be if it actually has the top left hand pixel of a character, and also has some idea about where the rest of the character field should be

The BSP algorithm depends upon findmg the border of the scanned m element To find a box around each area, or element, the BSP first traces the exterior boundary of the outermost black pixels Then it can start to deduce from the length of the boundary, something about the size and shape of the object found This will also distinguish noise from characters, given reasonable quality of image By starting this way, the BSP will segment the image mto objects

For reasonable quality prmted text, the BSP border tracing will trace and isolate the target area The BSP algorithms are defined m terms of the problem of findmg the border pixels of a connected block of pixels The BSP then regards the area as the set of pixels lymg mside this irregular box', as in Figure 10

First, it is assumed that the unage or area has a white (or different color) border around it, and the BSP has to find the bounding pixels of clumps of black pixels The BSP procedure starts at the top left of the unage at a white pixel, and scans horizontally until it encounters the end of the row or a black pixel In the former case, the procedure immediately proceeds to the start, at the left side of the next row of pixels down If it is at the bottom row already and there is no next row, the procedure has scanned the entire area, and it exits

If the procedure has found a black pixel, the procedure draws an imaginary line from the pixel just to the left of it, to the black pixel The procedure notes the direction of this lme, and the coordmates of the black pixel Then the procedure rotates the line, about the black pixel, m the positive direction (anticlockwise), until it meets another black pixel adjacent to the first black pixel, or returns to its oπgmal position In the latter case the black pixel is isolated, the procedure marks its location, deletes it and starts agam

If the procedure finds another black pixel then it is m one of the eight positions adjacent to the first black pixel Let the current black pixel the procedure is currently focused on, be pixel n When n= 1 the procedure is at the first black pixel, this also works for all of them

The procedure now moves to pixel n+1, notes its location, notes the direction from pixel n to pixel n+1, and takes the line joining pixels n to n+1 and rotates it about pixel n+1 in the positive direction, until it finds an adjacent black pixel which becomes pixel n+2 This process is repeated until pixel n+1 is a previously listed pixel and the line from pixel n to pixel n+1 is the same previously listed direction

Once a black object has been found, the pixel set of its border is determined by this algoπthm and this set of pixels together with every pixel (black or not) which is inside this border is stored as an object and then deleted from the area map, and the scan starts agam Only when the area has been totally blanked does the procedure terminate

To see if a pixel is inside the border, The algorithm draws a half ray starting at the pixel, and extends to the boundary It then counts the number of times this intersects the border If the number is even the starting pixel is outside, otherwise it is mside or on the border This works in almost all cases Where there is an anomalous area, the BSP algorithm can wobble the lme and the starting pixel a bit and then try agam If a pixel is inside the border and not on it, every adjacent pixel is either mside or on the border A floodfill' operation stoppmg at the border and starting inside it is used to label the pomts to be removed

If the unage has a black border, or a black box around a character, and if the border is broken, then the algoπthm will fail to find the desired set The BSP algorithm has special case code to account for these cases

The BSP algorithms allow for the processmg of highlighted textual data for many conceived purposes For the purpose of design, and immediate applicability, it is envisioned that BSP will be integrated mto (but not exclusively) the following families, or classes of product

• Contextual Question and Answer Applications,

• Applications that allow Editmg and Proofreading, and

• Specific Criteria Forms Processmg

Below, are examples of each of these applications of the BSP methods

The BSP technology will be part of a larger testmg or processmg application, where text-based queries are answered withm the same document areas For mstance, in the following example, as series of questions are laid out on a page in columnar form Multiple answers are also laid out in similar fashion

The form's applicable questions (there are more than one set of questions, used for random testmg) have been pre-colored (highlighted, as m Figure 11) by a test-giver, and then given to the test-taker, or applicant

The applicant then proceeds to mark up, or highlight (with a simple marker), the correct answers (as m Figure 12), and submits the test to a scanner for automatic grading In this case, where two highlightmg colors were used, one color was assigned to the questions and another to the answers

The novel part of this scenario is that discrimmation of the questions, along with the applicable answers, is done by scannmg the form, submitting the scanned data to the BSP application (along with the coloring criteria), having the BSP-based machme then submit the discriminated data areas to an OCR application, then grade or correct the test, without further operator intervention

It is withm the capabilities of the BSP system to take editor's submissions and on-the-fly-changes to editable material, and process the changes wholly by machme, without further human intervention Above, an editor's technical resources have setup the BSP system to accept scanned documents for editmg, with the following criteria

• All words or phrases marked with a "yellow" highlight should be changed, or re-visited by a junior editor as to whether they are semantically correct

• All words or phrases marked with a "red" highlight are to be stricken from the document

• All words marked with a "blue" highlight should be Capitalized

• All words marked with a "green" highlight must carry a trademark or copyright symbol

The BSP machme, m this case, may even be able to post-process the output, change capitalization, symbols, and look in a thesaurus for similar words to replace stricken ones with All without intervention, because of the colorized 'hints" given to the system by the initial use of simple markmg pens (Figure 13)

An example of this application, below, would be to submit inventory or order forms to the BSP system with colorized hmts as to what inventory items were to be dropped, to have their orders increased in quantity, or to order less of an item Agam, with subsequent OCR processmg, the BSP-enabled computational device could understand the color- cues change orders to third-party suppliers, or flag items for executive attention

As can be seen in Figure 14, below, the demarcated areas do not necessarily have to be perfectly rectangular This is because the BPS bounding area matchmg algorithms are designed to normalize a non-rectangular polygonal area If, for instance, an area is not evenly tall or wide, the procedures responsible for normalization of the area will automatically adjust the bounding rectangle to its maxima, in both dimensions

At an extrema, a demarcation may be faded, or geometrically uneven The normalization algorithms are as effective in dealing with uneven color, as they are in dealmg with geometry The coloration mdexmg algorithms compensate automatically, by utilizing mean sampling, for any variation in color or opacity, from adjacent pixel grouping, to pixel grouping By way of default, the handling of any vanation m size, coloration, or opacity/transparency, will be normalized before further processmg takes place

In the sample form below, one may see that the form is marked for processmg dependent upon contextual cuemg, with several different colors The BPS system will handle any arbitrary subset of color indices, dependent only upon available memory and other final end-product design constramts There is nothmg mherently more complex or difficult in having a 100 color mdexmg table, rather than a 5 color table The only difference will be dependent upon how fast the entire system must respond, smce processmg time will be elevated This is, of course, also dependent upon computational device processmg power, and what language/optimization is effected within the BPS module Smce the algorithms are language/processor mdependent, this is purely a product and marketmg-usability question, not one that is effected by the core BPS technology

Figure 15 shows in schematic, or program form, some of the steps comprising the present mvention

While the present mvention has been described with regards to particular embodiments, it is recognized that additional variations of the present mvention may be devised without departing from the inventive concept

INDUSTRIAL APPLICABILITY

It is an object of the present mvention to provide a system for discriminating text or other written communication including prmted and handwritten text, graphics, or otherwise

It is yet another object of the present mvention to provide a system for discriminating text that is easy to use It is yet another object of the present invention to provide a system for discriminating text that is based on highlighting, colorization, or similar visually-based and selectable aspects of text.

It is yet another object of the present invention to provide a system for discriminating text that may be used with either printed or electronic text.

It is yet another object of the present invention to provide means by which a student can indicate question and answer material for use in conjunction with a interactive computer-networked study aid or the like.

These and other objects, advantages, and the industrial utility of the present invention will be apparent from a review of the accompanying specification and drawings.

Claims

What is claimed is

A method for indicatmg and extractmg subject matter from a larger work, the steps comprising mdicatmg a first area in said larger work for extraction, and extractmg said first area from said larger work, whereby said first area may be made separately mampulable from said larger work

The method for mdicatmg and extractmg subject matter from a larger work as set forth in Claim 1, wherein said step of mdicatmg a first area further comprises mdicatmg said first area by highlightmg said first area

The method for indicating and extractmg subject matter from a larger work as set forth in Claim 2 wherem said step of indicatmg said first area further comprises indicating said first area by highlightmg said first area in a distinctive color

The method for indicatmg and extracting subject matter from a larger work as set forth in Claim 2, wherein said step of extracting said first area further comprises recognizmg said first area, copying said first area, and designating said first area as extracted from said larger work

The method for mdicatmg and extractmg subject matter from a larger work as set forth in Claim 4 wherein said step of recognizmg said first area further comprises recognizing said highlightmg as distinct from the larger work, and recognizing highlighted subject matter of said first area as distmct from said highlighting, and removmg said highlightmg from said subject matter of said first area

The method for indicating and extractmg subject matter from a larger work as set forth m Claim 5, further compπsmg transmitting said subject matter of said first area on for further use or processmg

A method of designating material from a larger work, the steps compπsmg calibrating a recognition system to associate highlight colors with corresponding categories highlightmg a document with said highlight colors to associate highlighted material m said document with said categories, discnmmatmg highlighted portions of said document, and extractmg said highlighted portions of said document, whereby the material designated by highlightmg may be obtamed and used separately from the larger work

The method of designatmg mateπal from a larger work of Claim 7, wherem the step of discnmmatmg highlighted portions of said document further comprises performing a fast Fourier transform on one of said highlighted portions, and constructmg a histogram representmg color versus a number of pixels havmg that color m said one highlighted portion