JP4936250B2 - Write extraction method, write extraction apparatus, and write extraction program - Google Patents

Description

  The present invention relates to a writing extraction method, a writing extraction apparatus, and a writing extraction program for extracting writing from a document with writing added.

  Since a document (paper document) written on a print medium such as paper has high readability and portability, a large amount of information is provided through the paper document. Also, writing actions such as marking a noticed portion of a paper document or writing a memo are common. Therefore, it can be said that valuable information can be obtained by extracting and analyzing the writing, even though the writing in the paper document has abundant information such as the user's interest and knowledge. Specifically, it can be used for a) browsing of writing by a user and b) identification of a rough position of writing.

  Considering the above application, it is not always necessary to accurately trace the writing area, and it is important that it is easy for humans to read. If an accurate extraction is attempted, the writing is often insufficiently extracted (for example, divided and extracted), and the readability by the user is deteriorated. Splitting also adversely affects the location identification. Conversely, extracting anything other than writing does not affect readability.

Until now, various methods (for example, nonpatent literature 1, 2, 3, and 4) which extract writing from a paper document have been proposed. These methods are intended for machine recognition and impose restrictions on the color and type of writing in order to extract writing with high accuracy. However, since there is no limitation on writing performed in a real environment, there is a need for a technique that can extract whatever writing is performed with any color.
More specifically, the above-described conventional methods are classified into the following two types.
The first type is not extraction of writing, but focuses on the use of extracted writing such as automatic calibration using writing (Non-Patent Documents 1 and 2). In these methods, since the extraction result is used by a machine, highly accurate extraction is required. In order to realize this, these methods place restrictions on writing. Specifically, the pen color that can be used for writing is determined in advance, and it is determined whether or not the writing is performed based on the color of the pixel of the scanned image. Therefore, these methods have a limitation in writing color.
The second type classifies connected components in an image into handwritten characters and printed characters (Non-Patent Documents 3 and 4). These methods have an advantage that writing and extraction can be performed only from an image of a written document. On the other hand, writing that can be extracted is limited to characters, and handwritten lines and figures cannot be extracted. In actual writing, figures such as underlines and arrows are frequently used, so a method capable of extracting such writing is required.

As a technique related to the present invention, the inventors have proposed a document image search method based on local arrangement of feature points (see, for example, Patent Document 1 and Non-Patent Document 5). In the following description, this proposed method is simply referred to as a document image search method. The relationship between the present invention and the document image search method will be described later.
International Publication No. 2006/092957 Pamphlet D. Mori and H. Bunke, "Automatic Interpretation and Execution of Manual Corrections on Text Documents", in Handbook of Character Recognition and Document Image Analysis, ed. H. Bunke and PSP Wang, pp.679-702, World Scientific, Singapore (1997). J. Stevens, A. Gee, and C. Dance, "Automatic Processing of Document Annotations", In Proc. 1998 British Machine Vision Conf., Vol. 2, pp.438-448 (1998). JK Guo and MY Ma, "Separating Handwritten Material from Machine Printed Text using Hidden Markov Models", In Proc. 6th International Conf. On Document Analysis and Recognition, pp.436-443 (2001). Y. Zheng, H. Li, and D. Doermann, "The Segmentation and Identification of Handwriting in Noisy Document Images", In Lecture Notes in Computer Science (5th International Workshop DAS2002), vol.2423, pp.95-105 (2002 ). Nakai, Kise, and Iwamura: "Use of affine and similarity invariants in high-speed document image retrieval using a digital camera", IEICE Technical Report, PRMU2005-188 (2006)

The present invention provides a technique that can accurately extract writing even when writing on a color paper document in an arbitrary color, or when writing a line or a figure in addition to characters. It is. More specifically, the present invention is based on the premise that an image (original image) of a paper document to be written is available, and obtains a difference between the document image (written image) including writing and the original document image. This is to extract writing.
In the present invention, the writing area is extracted extra so that the writing extraction does not become insufficient due to the above-described division. As a result, portions other than writing may be extracted while being mixed with writing, but it does not have a bad influence on the browsing and position identification by the user.

The present invention is a method for extracting writing from an image in which writing is added to a document image, and the original image before writing is added and the writing image after writing is expressed by color components. Each of the local regions of one of the images that are aligned with each other, the step of acquiring each as a set of local regions, the step of comparing the original image with the written image, Determining a local region of the pair by determining a similar local region from the pair of candidates, using a local region of the other image at a corresponding position or a position within a predetermined range as a pair candidate, A computer that extracts a local area that is included in the written image and that is not included in the original image as a writing based on the difference in color components of the local area Providing a write extraction method characterized.

  Further, from a different point of view, the present invention is an apparatus for extracting a writing from a writing image in which writing is added to a document image, the original image before the writing is added and the writing image after the writing is added. Are acquired as a set of local regions represented by color components, a registration unit that compares the original image and the written image and aligns both images, and both images that have been aligned For each local region of one of the images, the corresponding region or the local region of the other image at a position within the predetermined range is taken as a pair candidate, and the local component having the closest color component of the local region from among the pair candidates Included in the written image and included in the original image based on the difference between the color components of the paired local regions and the local region determining unit that determines each local region of the pair by determining the region Providing a write extraction device and a write extractor for extracting no local region as a write.

  Furthermore, from a different point of view, the present invention is a program for executing a process of extracting writing from a written image in which writing is added to a document image, and the original image and writing before the writing is added are added. An image acquisition unit that acquires the written image as a set of local regions represented by color components, a registration unit that compares the original image and the written image, and aligns both images, For each local region of one image of both images that have been matched, a local region of the other image at a position corresponding to it or a position within the predetermined range is set as a pair candidate, and a local region is selected from the pair of candidates. Based on the local area determination unit that determines each local area of the pair by determining the local area that has the closest color component and the difference between the color components of the paired local areas Included in, and provides a write extraction program characterized by executing the function of the write extraction unit for extracting a local region which is not included in the original image as a write to the computer.

In the writing extraction method of the present invention, for each local region of one image of both images, the corresponding region or the local region of the other image at a position within a predetermined range is used as a pair candidate. Since each local region of the pair is determined by finding the local region with the closest color component of the local region from the pair candidates, it is more accurate than when pairing local regions at corresponding positions in both images You can extract the writing. In other words, the corresponding local region can be accurately determined without strictly aligning the original image and the written image in units of local regions. Strict alignment in units of local areas is not necessary, and an accurate extraction result can be obtained without spending time for alignment.
In addition, since there is no restriction like the conventional method, even when writing in a color paper document with an arbitrary color, or when writing a line or figure in addition to characters, the writing is accurate. Can be extracted well.

  Here, the document image is an image in which information including characters and / or images is represented as an image. Further, the document image includes image data obtained by converting the image into electronic data.

  Writing refers to visible information added to a document image. Writing is an image composed of characters, line diagrams, and the like. In many cases, the writing is added by handwriting, but the writing is not limited thereto, and the writing may be added by other methods such as printing or stamping.

In addition, alignment means determining the geometric transformation parameters to be applied to one image in order to superimpose the original image and the portion existing before writing in the written image, and performing geometric transformation based on the determined parameters. A process applied to the target image. Geometric transformation includes projective transformation, affine transformation, similarity transformation, etc. depending on the degree of freedom of transformation. In the following embodiment, the alignment based on the similarity transformation having the lowest degree of freedom among the three types of geometric transformations is described. However, the essence of the present invention is not necessarily limited to this.
As a preferred aspect of the present invention, the local region may be a region having one or more pixels as a unit. Here, the image reading is performed in units of pixels. The local area may correspond to each pixel, or a plurality of pixels having a predetermined arrangement relationship may be used as a unit.
Preferably, the image reading is generally performed after being decomposed into each color component of red (R), green (G), and blue (B). That is, each process can be performed on the pixels expressed in the RGB color space. However, the present invention is not necessarily limited to this, and can be applied to pixels expressed in a color space other than the RGB color space. As other color spaces, the YMC color space, the Lab color space, and the like are known and can be converted into different color spaces by calculation.

Preferably, the step of extracting the local region as writing is a step of outputting the local region of the writing image of the pair when the difference between the color components of the paired local regions is larger than a predetermined threshold. May be. In this way, the density of the image extracted as writing is not affected by the value indicating the attribute of the local region of the original image, so that the extracted writing can be easily recognized.
Here, the value indicating the attribute of the local region corresponds to a value indicating the attribute, for example, when the local region is one pixel, the pixel value (the value of the color component of the pixel). When the local region is composed of a plurality of pixels, for example, a value obtained by averaging the pixel values corresponds to a value indicating an attribute. Alternatively, each pixel value may be averaged with a predetermined weight, or may be calculated by other procedures.

  Preferably, the step of aligning the images may include a process of performing color clustering on the original image and the written image, and obtaining correspondence between the original image and the written image for each color clustered image. In this way, the correspondence between the original image and the written image is taken for each color clustered image, so that even when the color original image is written in color, the writing can be extracted with high accuracy.

  Preferably, the step of aligning the images may include a process of extracting feature points from the original image and the written image, and taking correspondence between the original image and the written image for each extracted feature point. Good. In this way, since the alignment is performed using the feature points extracted from the original image and the written image, the alignment is performed compared to the case where the alignment is performed in consideration of the entire local region without using the feature points. The processing time required can be greatly reduced. That is, in order to realize the means for solving the problem, it is necessary to align the written image and the original image at high speed and accurately. For this problem, the document image search method described above can be used. The document image search method and the present invention are different from each other in the problem to be solved. However, according to the document image search method, it is possible to obtain a correspondence for each part between the document image of the search query and the searched document image in the search process. For example, FIG. 30 or FIG. 44 of Patent Document 1 shows an algorithm for search processing. If this algorithm is applied to the present invention, image alignment can be achieved at high speed and / or with high accuracy. be able to.

  Preferably, after the step of extracting the local region as writing, the computer further processes a shaping step of shaping and outputting the extracted writing, and the shaping step transmits the local region extracted as writing to the transmission unit. And a step of obtaining a mask pattern obtained by further expanding the area of the transmissive portion by a predetermined amount, and a step of superposing the obtained mask pattern on the original image and outputting an original image of a portion overlapping the expanded transmissive portion. May be included. In this way, more extracted local regions of writing can be extracted than when there is no shaping step, and writing can be recognized more easily. Depending on the threshold value used to determine the difference in color components, the line portion of the image extracted as writing may be interrupted or thinned. In this way, the line portion is prevented from being interrupted or thinned. Recognition of writing can be facilitated.

  Further preferably, the step of obtaining a mask pattern may include a step of binarizing each extracted local region of writing and a step of performing a connection process on each binarized local region. Good. By doing so, it is possible to repair the interruption of the line portion during writing.

Still preferably, the step of obtaining the mask pattern may further include a step of removing a region smaller than a predetermined area from the write region generated by the concatenation process. In this way, since dot-like noise can be removed from the extracted writing, writing can be recognized more easily than when this processing is not performed.
The preferred embodiments described above can be combined.

  In the following embodiments, it is assumed that the analysis of the extraction result is performed by visual observation, and a method is described in which better visibility is prioritized than strict extraction of writing. However, the essence of the present invention is not limited to this method. As a result of experiments, it was shown that good results can be obtained for document images containing many characters.

In the writing extraction method of the present invention, on the assumption that the original image can be acquired, the original image and the writing image are aligned, and writing is extracted by taking the difference. A feature of the present invention is flexible difference processing that is resistant to not only global shift but also local shift.
Further, in the above-described alignment, it is very preferable to obtain a geometric transformation parameter by searching for corresponding points and perform high-speed alignment using the document image search method.

Hereinafter, the present invention will be described in more detail with reference to the drawings. In addition, the following description is an illustration in all the points, Comprising: It should not be interpreted as limiting this invention.
1. Flow diagram 1 of processing is an explanatory diagram showing a flow of processing of the write extraction method of the present invention. The process is roughly divided into an alignment process 11 and a difference acquisition process 13.

In the alignment process 11, the written image is subjected to similarity conversion in accordance with the original image, and the difference in deviation, rotation, and scale is corrected. The alignment process is further divided into four small processes.
In the difference acquisition process 13, the corrected writing image is compared with the original image, and a local area that is included in the writing image and not included in the writing image is output as writing by taking the difference. In the following description, a case where the local region is one pixel will be described as a representative example. The difference acquisition process is divided into six small processes.

2. Alignment
2.1. Feature point extraction The first process in position alignment is feature point extraction. Here, points that are robust (strong) against deformation and noise are extracted as feature points from the written image and the original image. In this embodiment, the feature point is the centroid of the connected component for each color obtained by color clustering the image in the RGB color space. This is intended to use the position of a monochromatic area such as a character as a feature point. Characters have a characteristic arrangement, and the contrast with the background is large, so that stable extraction is possible.
Specific processing will be described below. First, the image is reduced to reduce processing. As the reduction ratio, the same value L is used for both the written image and the original image. Then, in order to reduce the influence of noise, the image is smoothed by a morphological operation. Specifically, the erosion calculation with 3 × 3 rectangular elements is repeated l times. Next, after reducing the image by X times, the image is clustered into k color clusters with pixel colors by the k-means method. Here, the number k of color clusters is a predetermined value. Further, the clustering repetitive process is terminated when the number of processes exceeds i or when the moving distance of the center of gravity is less than p.

  Next, as shown in FIG. 2, the target image 15 is decomposed into k = 5 color cluster images 17a, 17b, 17c, and 17d based on the result of the color clustering. A connected component is extracted from the image of each color cluster obtained in this manner, and the area is examined to remove those having an area of E or more and those having an area of e or less. This is because a small area due to noise or the like or a large area such as the background is considered to be unstable. The centroid of the connected component remaining in this way is used as a feature point for each color cluster.

  The k-means method is a known technique for classifying given data into optimal clusters when given the final number of clusters (number of groups) k and evaluation criteria for each cluster (for example, Supervised by Mikio Takagi and Yoshihisa Shimoda, Image Analysis Handbook, 1st Edition, University of Tokyo Press, September 10, 2004, p.1576-1579).

2.2. Corresponding point search The second process is a corresponding point search. Here, the feature points of the written image and the feature points of the original image are associated by applying the document image search method.

  At this stage, each of the written image and the original image has a feature point for each of k color clusters. Since there may be many noise feature points depending on the color cluster, it is easier to obtain the correct corresponding points by associating the feature points of all the clusters, instead of associating them together. Therefore, prior to the association between the feature points, the one-to-one association between the color cluster of the written image and the color cluster of the original image is performed. In this embodiment, the color clusters are associated in ascending order of the centroid distance in the RGB color space. Specifically, the following processing is performed. First, distances are obtained for all combinations of the color cluster of the written image and the color cluster of the original image. Next, the pair with the smallest distance is associated. Then, the pairs with the smallest distances are associated in order. However, if any color cluster in the set has already been associated, the association is not performed. This process is repeated until the correspondence between the color clusters of all written images and the color clusters of the original image is determined on a one-to-one basis.

Next, feature points are associated for each set of associated color clusters. This is because, as in the document image search method, the feature points of the original image are registered in the database, and the process of searching for the corresponding features for each feature point of the written image is associated with each set of color clusters. It is realized by doing for. However, it differs from the document image search method in that only one image is registered. Note that the parameters used in the corresponding point search are the range n of neighboring points in which the feature amount calculation is performed, the number m of points used in the feature amount calculation, and the discretization level number d of the feature amount.
With the above processing, the correspondence relationship between the feature points is obtained for each set of color clusters. The correspondence of the entire image is obtained by collecting the correspondences of all the color clusters. FIG. 3 shows the correspondence between feature points.

2.3. Acquisition of similarity transformation parameters The third process is acquisition of similarity transformation parameters. Here, the similarity conversion parameter is acquired based on the correspondence between the feature points. The similarity transformation parameter is a four-dimensional vector composed of elements of scaling, rotation, and translation in the x and y axis directions. Similar transformation parameters are calculated from two sets of corresponding points. However, since the corresponding points in this embodiment may include wrong ones, it is necessary to calculate the conversion parameters by removing the wrong corresponding points. Therefore, the conversion parameter is estimated using RANSAC. RANSAC is a well-known technique often used to define the correspondence of feature points between multiple images (eg, MAFischler and RCBolles, "Random Sample Consensus: A Paradigm for Model Fitting with Application to Image Analysis and Automated Cartography"). , Comm. ACM, Vol. 6, No. 24, pp. 381-395 (1981)).

  A conversion parameter estimation algorithm is shown in FIG. First, two correspondences are selected at random from the set of correspondences (second line in FIG. 4). Then, a similarity transformation parameter is obtained from the two correspondences (third line). Next, the obtained similarity transformation parameter is evaluated. This is performed by setting the number of correspondences supporting the similarity transformation parameter as a score (line 4). Specifically, the point on the writing image side of the correspondence is converted by the parameter, and the distance between the coordinates of the obtained point and the coordinates of the point on the original image side of the correspondence is obtained. Assume that the relationship supports the parameter. Such processing is repeated T times (for loop in the 1st to 6th lines), and if the maximum score is equal to or greater than the threshold (7th line), the parameter having the maximum score is used as the estimation result (8th line). . Otherwise, the estimation failure is assumed (line 10).

2.4. Similarity Conversion Process The fourth process is a similarity conversion process. Using the similarity transformation parameter obtained so far, the similarity transformation is applied to the written image, and the original image is aligned. As a result, a corrected written image having the same size as the original image is obtained.

3. Difference acquisition
3.1. Preprocessing As described above, the difference acquisition process 13 is divided into six small processes. The first process is a preprocess. Here, image processing is performed on the corrected written image and the original image in order to appropriately perform the subsequent processing.
First, to reduce processing, the written image and the original image are reduced to Z times and z times, respectively.
Next, a Gaussian filter having a kernel size of g × g is applied to the written image, and the dot pattern generated when printing with the printer is removed. This is because in an electronic document, a portion that is an intermediate color becomes a dot pattern that uses a primary color during printing, and noise is generated if the difference is taken as it is.

  Processing is also performed on the original image. Here, two patterns are considered as the original image. One is an image obtained directly from an electronic document, and the other is an image obtained by printing an electronic document with a printer and capturing it with a scanner. In the former case, the image is fattened by repeating erosion calculation R times. This is to appropriately obtain a difference from a written image that is blurred during printing. In the latter case, the dot pattern is removed by applying a G × G Gaussian filter as in the case of the written image.

3.2. Difference / Threshold Processing The second processing is difference / threshold processing. Here, the corrected written image and the original image are compared for each pixel, and difference and threshold processing are performed. However, there may be a slight deviation between the written image and the original image due to distortion at the time of image capture by the scanner, error of feature points, and the like. For this reason, it is necessary to take the difference while considering the deviation.

  In this embodiment, the difference and threshold processing is performed as follows. First, each pixel of the written image is compared with the corresponding pixel of the original image. The pixels of the original image to be compared with the pixels of the written image are not of the same coordinates. In this embodiment, as shown in FIG. 5, pixels of the original image in an N × N square area centered on the same coordinates are candidates for corresponding pixels. Among the pixels in the region, the pixel closest in pixel value to the pixel of the writing image to be compared is set as the corresponding pixel. In this way, by searching for the closest pixel value from the pixels within the predetermined range, it is possible to accurately find the corresponding pixel even if there is a slight deviation between the written image and the original image. The difference between the pixel values of the written image and the original image obtained in this way is obtained.

  Here, if the pixel value difference is output as it is, the writing made on the colored background in the subsequent binarization processing may be erased. For example, when writing is performed on a dark background, the difference value becomes small and may be lost depending on the binarization threshold. If a threshold value for binarization is set so that even a small difference is extracted as writing, dirt or discoloration appears as noise this time. In order to avoid such a problem, in this embodiment, the difference between pixel values is compared with a predetermined threshold value. When the threshold value is exceeded, the pixel value of the written image is output as it is, not the difference in pixel value. Thereby, only dark color writing can be obtained.

3.3. Binarization The third process is a binarization process. Here, as shown in FIG. 6, the writing (difference image) obtained by the difference / threshold processing is binarized with a predetermined threshold. As the binarization threshold, u is used for an image obtained by a scanner, and x is used for an image obtained directly from an electronic document. In general, u is less than x. The reason is as follows. Since the written image is captured by the scanner, when the original image is captured by the scanner, since both have undergone the same image conversion, there is almost no difference in the area other than the writing. On the other hand, when the original image is obtained by converting from an electronic document, a difference occurs even in an area other than writing because of a difference in color. In order to remove such a difference that is not an extraction target, a larger threshold is required in the binarization process.

3.4. Write Concatenation Process The fourth process is a write concatenation process. In the process so far, the difference between the written image and the original image is obtained for the extraction of writing, and binarization is performed. The problem with the processing so far is that writing is partially lost due to noise. Therefore, writing is restored by collecting connected components by image processing using closing (closing) which is one of the morphological operations.

  FIG. 7 shows a connection process of writing by closing. Closing means that the connected component is first expanded by dilation, and then the connected component is reduced by erosion. Here, since the number of pixels expanded / reduced in each process is the same, the area of isolated connected components such as sesame salt noise (dot noise) does not change. On the other hand, in the divided connected components as shown in FIG. 7, since the connected components are combined by dilation, the combined components are maintained even after being reduced by erosion. As a result, the divided writing can be linked. Note that the number of repetitions used in the closing process is h.

  The area (transmission area) obtained by the above processing corresponding to writing becomes larger than the original writing. This is for use as a mask in the subsequent AND processing. In order to prevent writing loss in AND processing, the transparent area of the mask needs to be larger than writing.

3.5. Noise removal The fifth process is a noise removal process. Here, fine noise generated by the difference / binarization processing is removed. Specifically, as shown in FIG. 8, first, the area of each connected component is examined. Then, only connected components having an area equal to or larger than a predetermined threshold value M are written out. Thereby, noise is removed. Even if the write area is finely divided at the stage of the difference / threshold process and the binarization process, the area coupled to the area of the threshold value M or more in the subsequent write connection process is erroneously removed by this noise removal process. Never happen.

3.6. AND Process The final sixth process is an AND process. A dilation operation is applied to the image obtained by the above processing with the repetition number D to create a mask. Writing is extracted by taking the AND of the mask and the writing image for each pixel. The process is shown in FIG. As described above, since the transparent region of the mask is made larger, not only the writing but also the background is extracted. However, when visual analysis is performed, writing and background can be easily discriminated, which is not a problem.

4. Experimental Examples Experimental examples and results of the method of the present invention will be described below. In this experimental example, an image obtained from a color PDF file was used as an original image, writing was performed on a printed PDF file, and an image obtained by a scanner was used as a written image.
The experiment was performed using two original images. One is an image obtained by converting a PDF file into a raster format (original image A), and the other is a PDF file that is once printed and captured by the same scanner used to acquire the written image (original image) B). Since the former is not subjected to color change or distortion during printing or scanning, the corresponding point search process and difference process with the written image are relatively difficult. Since the latter causes the same change in both the original image and the written image, these processes are relatively easy. The number of original images is 109 for each.

  Written images were created by writing characters and figures in black / red / blue ballpoint pens at several locations on a printed document and capturing them at 600 dpi using a scanner. Since three kinds of writing images are prepared for one original image, the number of writing images is 327. Examples of two types of original images and written images are shown in FIG. 10A shows the original image A, FIG. 10B shows the original image B, and FIG. 10C shows the written image.

  The computer used in the experiment is equipped with AMD Opteron 2.8GHz CPU and 16GB memory. The experimental parameters are shown in Table 1. Here, each alphabet in Table 1 corresponds to the description of the embodiment described above. For example, L is an image reduction magnification in feature point extraction. Further, l is the number of repetitions of erosion calculation in feature point extraction.

4-1. When using images from PDF

  First, the case where an image from a PDF file is used as the original image will be described. The parameters in this experiment are shown in Table 1. For the original image, 109 PDF files converted to images at 600 dpi were used. Since three kinds of writing images are prepared for one original image, the number of writing images is 327. Table 2 shows the result of visual evaluation of the written extracted image.

Here, success, fading / noise, and failure indicate that the extracted writing is in the following state.
1) Success: Writing is extracted sufficiently and there is almost no noise (see Fig. 11)
2) Fading and noise: Writing is partially faint or noticeable noise (see Fig. 12)
3) Failure: Writing has been almost completely lost or noise has occupied most of the paper (see Fig. 13).
Table 2 shows an image including many monochromatic areas such as characters and an image having few monochromatic areas. This is for clarifying the influence of the nature of the input image in the embodiment in which the center of gravity of the single color connected component is the feature point. Whether or not there are many monochromatic regions was determined by visual observation.

  Note that FIG. 11A shows a writing image classified as “success”, and FIG. 11B shows writing extracted from the image of FIG. FIG. 12A shows a writing image classified as “having blur and noise”, and FIG. 12B shows writing extracted from FIG. 12A. FIG. 13A shows the writing image classified as “failure”, and FIG. 13B shows the writing extracted from FIG. 13A.

  The experimental results will be discussed below. It can be seen that the success rate is lower when images from PDF are converted directly into images than when scanned images are used. This is considered to be because the original image has not undergone printing and scanning, and the hue of the original image and the written image is different, so that the corresponding point search fails. If the hues of the original image and the written image are different, the result of color clustering in the feature point extraction process will be different, and therefore, corresponding feature points may not be obtained. In addition, the correspondence between the color clusters may fail in the corresponding point search process. This is considered to be the reason why the failure rate is high when an image from PDF is used as the original image.

  This problem can be dealt with by creating a model of image degradation in printing and scanning. If the original image was obtained from PDF, it would be possible to reduce the difference in hue and reduce failure in corresponding point search if processing that reproduces the same deterioration as the written image can be added to the original image. .

  Also, from Table 2, it can be seen that a higher accuracy was obtained for an image including a large number of monochrome regions. This is because an image with a small monochromatic area is not suitable for the method of this embodiment. In this embodiment, the feature point is the center of gravity of the connected component as a result of color clustering. This is suitable for an image including a large number of single-color figures such as characters as shown in FIG. However, as shown in FIG. 13, feature points cannot be stably extracted in an image that does not include so many characters and occupies most of a gradation figure or photograph. That is, different feature points are extracted between the original image and the written image. Therefore, the corresponding point search cannot be performed correctly, and subsequent processing fails. From this, it can be expected that higher accuracy can be expected for a character-centered monochrome document in which feature points are stably obtained.

Regarding the processing time, when an A4 size document was acquired at a resolution of 600 dpi, an average of about 20 seconds per sheet was obtained, and for an A3 document, about 40 seconds were obtained. Most of the processing time is spent on image processing such as feature point extraction and difference processing, and it is thought that processing time can be shortened by simple high-speed processing such as reducing the resolution.
4-2. Using scanned images

Next, a case where a scanned image is used as the original image will be described. In this experiment, the original image was obtained by acquiring the same scanner as the written image and a document without writing with the same settings.
In order to clarify the problems in applying the method of the present invention, a color document containing many figures and photographs such as posters and web pages as shown in FIG. 14 (a), and FIG. 14 (b) Each experiment was carried out using a monochrome document in which the letters shown occupy most of the paper.
4.2.1. Experiments with color documents

In this experiment, 109 original images and 327 written images were used in the same manner as in the experiment using images from PDF. The same parameters as in Table 1 were used except that the maximum area E of the connected components during feature point extraction was ∞ (infinity). The reason why E is set to a very large value is to extract feature points from a large connected component.
In this experiment, not only the final writing extraction result but also the corrected writing image at the time when the alignment process was completed were evaluated. This is to clarify the respective performances of the alignment process and the difference process. The determination of the result of the alignment process was performed visually.

The alignment results are shown in Table 3, and the extraction results are shown in Table 4.

  The experimental results are considered below. First, consider the alignment results shown in Table 3. Overall, 91% were successfully aligned. In particular, 99% succeeded in those with many monochromatic areas. FIG. 15 is an explanatory diagram illustrating an example of corresponding points when there are many single color regions. As described above, when there are many monochromatic regions, many stable feature points can be extracted, and many correct corresponding points can be obtained. On the other hand, FIG. 16 is an explanatory diagram showing an example of corresponding points when there are few monochromatic regions. If the monochromatic area is small, correct corresponding points cannot be obtained. Even those with many monochromatic areas failed in two images. These have a plurality of arrangements of the same feature points such that the same text exists in a plurality of places in the same image. FIG. 17 is an explanatory diagram showing an example in which the same feature points are arranged at a plurality of locations. In such a case, the alignment fails because an incorrect response occurs. Overall, 91% of color documents such as posters were successfully registered, indicating that the method of this embodiment is robust.

  Next, consider the results of write extraction shown in Table 4. In total, 76% succeeded in extracting and writing out only 88% of the monochromatic areas. Also, in applications such as simply browsing, there is no problem even if there is a slight blur or noise. In such a case, the success rate is 91% overall, and it can be said that it is 98% in the case of many monochromatic regions. Most of those classified as failures are those that failed at the time of alignment. Even though the alignment was successful, those that were classified as failed or noise / blurred were written on a colored background, so the writing was lost in the difference processing, and there was a slight error in alignment. The noise is generated after the difference processing.

4.2.2. Experiments using monochrome documents Experiments using color documents showed that writing extraction is difficult for images with few monochromatic areas, such as text, and images written on a colored background. In this experiment, the technique of this embodiment was applied to a monochrome document including a large number of characters and written on a white background, and the performance of an object suitable for the technique of the present invention was examined.
In the experiment, we used 34 original images created from PDF files of English papers and 34 written images written on them with black, red, and blue ballpoint pens. The number of clusters at the time of color clustering was set to k = 2, and other conditions were the same as in the case of item 4.2.1.

  The results of alignment and writing extraction are shown in Table 5 and Table 6, respectively. As shown in Table 5, in the case of a monochrome document, alignment was successful for all images. This is because a large number of characters are included, so that it is possible to obtain many stable feature points, and thereby it is possible to estimate an appropriate conversion parameter from many correct corresponding points. As shown in Table 6, blurring occurred in the extracted writing in the two documents. These have been partially lost in the differential processing because the writing with the red ballpoint pen was thin. However, 94% of them succeeded in write extraction, and none failed. From the above, it has been shown that the method of the present invention is extremely effective for a character-centered monochrome document.

  As a technique for improving the feature point extraction process so that stable feature points can be obtained even for objects that do not contain many characters, Harris operators (for example, C. Harris and M. Stephens, "A Combined Corner and It is considered effective to introduce research results in computer vision such as Edge Detector ", Proc. 4th Alvey Vision Conf., Pp. 147-151 (1988)). Further, it is considered that further improvement in image processing is possible.

  In addition to the embodiments described above, there can be various modifications of the present invention. These modifications should not be construed as not belonging to the scope of the present invention. The present invention should include the meaning equivalent to the scope of the claims and all modifications within the scope.

In accordance with the present invention, writing is extracted along with information about position, size, and shape within the document. When the original electronic document is available, information such as position, size, and shape in the document can be used to determine what figure, word, and character exist around the writing. By using this information, writing can be indexed. Specifically, words existing around the writing are given to the writing as keywords. Once a keyword is assigned, it is possible to search for a keyword for writing, and (1) it is possible to extract information about which part of the document the person who wrote is interested in (user profile Creation), and (2) which part of the document is important can be ranked (for example, the places underlined and enclosed by many users are important).

It is explanatory drawing which shows the flow of a process of the writing extraction method of this invention. It is explanatory drawing which shows an example of a target image and the image obtained by color-clustering it. It is explanatory drawing which shows the example with which matching with the original image and the writing image was made | formed in the feature point unit of the color cluster. It is explanatory drawing which shows an example of the algorithm which estimates a similarity transformation parameter based on the correspondence of a feature point. It is explanatory drawing which shows the procedure of the process which calculates | requires the difference of the corresponding pixel of an original image and a writing image. It is explanatory drawing which shows a mode that the write extracted by the difference and threshold value process is binarized. It is explanatory drawing which shows the procedure of the connection process of the writing by closing. It is explanatory drawing which shows the procedure of a noise removal process. It is explanatory drawing which shows the procedure of AND processing. It is explanatory drawing which shows an example of two types of original images used for experiment, and a writing image. In an experimental example, it is explanatory drawing which shows the write image classified into "success", and the extraction result of writing. It is explanatory drawing which shows the extraction result of the writing image classified into "with a blur and noise" in an experiment example, and writing. In an experimental example, it is explanatory drawing which shows the writing image classified into "failure", and the extraction result of writing. It is explanatory drawing which shows the example of the original image different from FIG. 10 used for experiment. It is explanatory drawing which shows the example of a corresponding point when there are many monochrome areas. It is explanatory drawing which shows the example of a corresponding point when there are few monochrome areas. It is explanatory drawing which shows the example which has arrangement | positioning of the same feature point in several places.

Explanation of symbols

11 Registration processing 13 Difference acquisition processing 15 Target image 17a, 17b, 17c, 17d, 17e Color cluster image

Claims (9)

A method for extracting writing from an image in which writing is added to a document image,
Acquiring each of the original image before the writing is added and the writing image after the writing is added as a set of local regions represented by color components;
A step of aligning the two images by comparing the local region of the local region and writing the image of the original image,
For each pixel of one image of both images that have been aligned, a pixel corresponding to the other image or a pixel in the other image at a position within a predetermined range is used as a pair candidate, and similar pixels are selected from the pair candidates. Determining a pair of pixels by determining;
A writing extraction method characterized in that a computer processes a step of extracting pixels included in a writing image and not included in an original image as writing based on a difference in color components of a paired pixel . Step, if the difference in color components of pixels in the pair is greater than a predetermined threshold, the method of writing the extraction according to claim 1, which is a step of outputting a pixel of the written image of the pair of extracting pixels as write .   The writing extraction method according to claim 1, wherein the step of aligning the images includes a process of performing color clustering on the original image and the written image and obtaining correspondence between the original image and the written image for each color clustered image.   The writing extraction method according to claim 1, wherein the step of aligning images includes a process of extracting feature points from the original image and the written image, and taking correspondence between the original image and the written image for each extracted feature point. . After the step of extracting pixels as writing, the computer further processes a shaping step of shaping and outputting the extracted writing,
The shaping step includes obtaining a mask pattern in which a pixel extracted as writing is a transmissive portion, and the region of the transmissive portion is further expanded by a predetermined amount;
The write extraction method according to claim 1, further comprising the step of superimposing the obtained mask pattern on the original image and outputting the original image of a portion overlapping the expanded transmission part. The step of obtaining the mask pattern includes the step of binarizing each pixel of the extracted writing,
The writing extraction method according to claim 5, further comprising a step of performing a connection process on each binarized pixel . The write extraction method according to claim 6, wherein the step of obtaining a mask pattern further includes a step of removing a region smaller than a predetermined area from write regions generated by the concatenation process. An apparatus for extracting writing from a written image in which writing is added to a document image,
An image acquisition unit for acquiring the original image before the writing is added and the writing image after the writing is added as a set of local regions represented by color components, and
By comparing the local region of the local region and writing the image of the original image and the positioning unit for aligning the two images,
For each pixel of one image of both images that have been aligned, a pixel corresponding to that position or a pixel in the other image at a position within a predetermined range is used as a pair candidate, and the color component of the pixel from the pair of candidates A counter- pixel determining unit that determines a pair of pixels by obtaining the closest pixel ;
A writing extraction apparatus comprising: a writing extracting unit that extracts pixels included in a writing image and not included in the original image as writing based on a difference between color components of the paired pixels . A program for executing a process of extracting writing from a writing image in which writing is added to a document image,
An image acquisition unit for acquiring the original image before the writing is added and the writing image after the writing is added as a set of local regions represented by color components, and
By comparing the local region of the local region and writing the image of the original image and the positioning unit for aligning the two images,
For each pixel of one image of both images that have been aligned, a pixel corresponding to that position or a pixel in the other image at a position within a predetermined range is used as a pair candidate, and the color component of the pixel from the pair of candidates A counter- pixel determining unit that determines a pair of pixels by obtaining the closest pixel ;
A computer is caused to execute a function as a writing extraction unit that extracts pixels included in a writing image and not included in the original image as writing based on a difference between color components of the paired pixels. Write extraction program.