JP2014048819A - Image processor, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To properly eliminate a handwritten part of a document image by using a handwritten image generated from coordinate data by an electronic pen.SOLUTION: An image processor includes: image acquisition means for acquiring a document image of a document including handwriting by an electronic pen; handwriting acquisition means for acquiring coordinate data of the handwriting by the electronic pen; generation means for generating a plurality of handwritten images based on the coordinate data by using a plurality of different line width; retrieval means for retrieving an image of a handwritten part similar to the handwritten image from the document image in each generated handwritten image; selection means for selecting a handwritten image that is the most similar to the image of the handwritten in the document image; and elimination means for eliminating the image of the handwritten part from the document image by using the selected handwritten image.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for processing document image and handwriting coordinate data.

  In recent years, there is an increasing number of cases where paper documents are converted into images by a scanner or the like and stored and reused as electronic documents. One reason for this is the reduction in paper storage volume. The digitization of paper documents makes it easy to store and carry many documents.

  Also, by recognizing an electronic document and performing a search using a character code, there is an advantage that a desired document can be easily found and the reuse of the material is easier than when using a bundle of thick paper materials. .

  When paper materials distributed at a meeting or the like are scanned and digitized, handwriting written during the meeting may be unnecessary. For example, when a distributed material is shared by a plurality of people and used, a document image without writing is referred to. Since important information is also included in writing itself, a document image with writing may be required. Which one is required depends on the user's intention and purpose of use.

  If a document image before writing is necessary, the document image may be generated by scanning before writing on the material. However, since the material distributed at the meeting or the like is actually used at the meeting immediately, it often takes time to scan and create a document image before the meeting. In addition, an environment (such as a scanner device) that can immediately scan a paper document at a meeting place is not always available.

  Therefore, when the paper material is scanned, the written handwriting already exists, and the document image in the state before writing may not be obtained.

  In addition, in order to be able to search the contents of materials distributed at a meeting or the like later, paper materials may be read by a scanner, characters may be recognized from a document image, and keywords may be registered. At this time, if there is writing or the like on paper material, recognition of the line or the like may be wrong, and character recognition may not be performed properly.

  As described above, when a paper material is digitized by a scanner or the like, there is a need to erase a handwritten handwritten (added) handwriting. As a technique for erasing the handwriting image, for example, there is a technique for detecting and erasing the handwriting image from the scan image using the character stroke thickness and linearity, the regularity of the character spacing, the frequency component distribution of the character string image, etc. is there. However, when there are handwritten characters or the like in the original paper material, the handwritten handwritten portion that has been additionally written cannot be detected properly, and the handwritten image cannot be erased appropriately.

  On the other hand, there are electronic pens (or digital pens) that can acquire handwriting as coordinate point data. The electronic pen is a writing device that can acquire a series of coordinate point data (x coordinate value and y coordinate value) of written coordinates simultaneously with writing on paper. Since ink comes out from the pen tip in the same way as a normal pen, characters and figures are written on the paper, but at the same time, the coordinate point data of the handwriting can be acquired. Hereinafter, the coordinate point data of the handwriting is also simply referred to as coordinate data.

  There are several types of electronic pens. For example, a pen using an ultrasonic transmitter, a paper placed on a tablet device, a normal stylus pen attached to the tip of a stylus pen, and a special paper with minute dots. There are anotopens to use.

JP 2002-082662 A Japanese Patent Application Laid-Open No. 2002-082664 JP 2008-102782 A JP 2007-079506 A

  Here, when the handwriting part is erase | eliminated from a document image using the coordinate data acquired from the electronic pen, there exists a problem that the reliability regarding the information regarding the writing position which an electronic pen has is low.

  For example, during the meeting, the material is referred to before and after and writing is performed as appropriate, so the reliability of the page information related to writing that the electronic pen has is low. Further, when placing paper on the tablet device, if the paper is not set correctly at a position relative to the paper fixing position, an error occurs in the measurement position of the electronic pen.

  Therefore, the information regarding the writing position of the electronic pen is low in reliability, and it is difficult to use the information regarding the writing position for erasing the handwriting.

  Accordingly, the disclosed technique aims to provide an image processing apparatus, an image processing method, and a program that can appropriately erase a handwriting portion of a document image using a handwriting image generated from coordinate data by an electronic pen. And

  An image processing apparatus according to an aspect of the disclosure includes an image acquisition unit that acquires a document image of a document including a handwriting by an electronic pen, a handwriting acquisition unit that acquires coordinate data of a handwriting by the electronic pen, and a plurality of different line widths. A generating unit configured to generate a plurality of handwriting images based on the coordinate data; a searching unit configured to search for an image of a handwriting part similar to the handwriting image for each generated handwriting image; and the document image Selecting means for selecting the handwriting image most similar to the image of the handwriting portion, and erasing means for erasing the image of the handwriting portion from the document image using the selected handwriting image.

  According to the disclosed technique, it is possible to appropriately erase the handwriting portion of the document image using the handwriting image generated from the coordinate data by the electronic pen.

The figure explaining the flow of the handwriting erasure | elimination process of base technology. The figure explaining the specific procedure of generalized Hough conversion. The figure which shows an example which produces | generates a handwriting image from coordinate point data. The figure which shows an example of the edge image produced | generated from the thin handwriting and the thick handwriting. The figure which shows the difference in the thinning image by the thickness of a character. The block diagram which shows an example of the hardware of the image processing apparatus in an Example. The block diagram which shows an example of the function of the image processing apparatus in an Example. The figure explaining the flow of the handwriting erasure | elimination process in an Example. The figure which shows an example of the data structure of coordinate data. The figure explaining the meaning of a writing state. The figure which shows an example of the state which connected the coordinate point with the line segment. The figure which shows an example of the state which filled between the coordinate points with 1 pixel. The figure which shows an example of the state as which the boundary of the handwriting was determined. The figure which shows an example of the state which filled the pixel with line | wire width w. The figure which shows the operation example of a search. The figure which shows an example of the search result for every line width. The figure which shows the example painted simply with a background color. The figure explaining the filling process by an Example. The figure explaining the filling color determination. The flowchart which shows an example of the handwriting deletion process in an Example. The flowchart which shows an example of the process which produces | generates a some handwriting image. The flowchart which shows an example of the pixel filling process after handwriting deletion.

  First, a technique that is a premise of the embodiment will be described. In the base technology, a handwriting image is generated from coordinate data, an image close to the handwriting image is searched from a document image read from a scanner, and the searched partial image is erased based on the handwriting image.

  FIG. 1 is a diagram for explaining the flow of the handwriting erasing process of the base technology. As shown in FIG. 1, the electronic pen 10 is used to write on the material by appending. At this time, the electronic pen 10 acquires coordinate data written on the material.

  The scanner device 20 reads a material added by the electronic pen 10 and acquires a document image a101 that is a scanned image. Further, the device that has acquired the coordinate data from the electronic pen 10 generates the handwriting image a102 from the coordinate data.

  The apparatus that generated the handwritten image a102 acquires the document image a101 from the scanner apparatus 20, and searches for a partial image similar to the handwritten image a102 in the document image a101. The apparatus can generate the document image a103 of the original material by erasing the handwriting from the retrieved partial image using the handwriting image a102. The original material refers to a material that includes characters, figures, and the like that were originally written excluding the part written with the electronic pen 10.

  Here, the coordinate values that can be acquired by the electronic pen 10 are relative coordinates with respect to a reference point that the electronic pen 10 has. Coordinate data on the paper surface can be obtained by carefully matching the origin of coordinates and the position of the paper.

  However, since the scanned document image has minute fluctuations in the position and angle of the paper at the time of scanning, the coordinate system of the document image and the coordinate system of the coordinate data do not necessarily coincide completely. Even if the position of the paper is shifted by several millimeters, the handwriting pixels and the handwriting coordinate values are completely different. Therefore, even if the coordinate values that can be acquired with the electronic pen 10 are handled as absolute coordinates, it is not possible to erase only the pixels belonging to the handwriting from the scanned image.

  That is, the position of the handwriting obtained using the electronic pen 10 means that the rough position and shape of the handwriting can be understood, and the accuracy of the coordinates is insufficient for the purpose of erasing only the handwriting from the scanned image. is there.

  In addition, the normal electronic pen 10 cannot acquire information that the writing paper has been changed to another paper when the paper is turned and changed. Even if the page movement timing is specified by providing a dedicated button or the like, pages are not always referred to in order when referring to a plurality of pages of material. For example, it is possible to return to the first page immediately after writing on the tenth page.

  Therefore, it is difficult to reliably record on which page the coordinate data is written, which is troublesome for the user. Therefore, the coordinate data obtained by the electronic pen 10 cannot be interpreted as absolute coordinates with respect to the origin of the scan image, and alignment processing is required.

  Therefore, attention is focused on an arbitrary figure search technique using generalized Hough transform. For generalized Hough transform, see “D.H. Ballard,“ Generalizing the Hough transform to detect arbitrary shapes, ”Pattern Recogniton, vol.13, no.2, pp.111-122, 1981.”.

In the generalized Hough transform, an edge point is obtained from both an image to be searched and an image to be searched. In this case, the image to be searched is a handwriting image, and the image to be searched is a scanned image. An arbitrary figure is searched for by voting the position and direction of the obtained edge point. FIG. 2 is a diagram for explaining a specific procedure of the generalized Hough transform.
(1) Edge points are detected from both the search image and the target image. (2) The score is voted relative to the reference point from the direction of all edge points of the target image. (3) The vote value in the search image. The maximum value of the image indicates the position of the reference point of the search image. Usually, the edge point is determined by the Sobel operator, and the gradient value of the gradient value exceeding a certain threshold is stored. As an example, refer to Japanese Patent Application Laid-Open No. 09-245167.

  FIG. 3 is a diagram illustrating an example of generating a handwriting image from the coordinate point data. Since the handwriting to be searched is given by the coordinate point data of the handwriting, a handwriting image may be generated from the coordinate point data and searched as shown in FIG. 3 (see, for example, JP 2011-258129 A). Further, since the coordinate point data can be regarded as a set of line segments obtained by approximating the curve of the handwriting with a broken line, there is a method in which the direction of the handwriting is obtained from each line segment and regarded as the edge direction.

  Here, when searching for the handwriting included in the scanned document image using the coordinate point data of the same handwriting, the thickness of the handwriting is not known, and thus the search accuracy may be reduced. The handwriting written on the paper document has a different thickness depending on the electronic pen 10 used and how the user writes. For example, there is a large difference in line thickness between a thin ballpoint pen and an aqueous sign pen. On the other hand, since the handwriting to be searched is the coordinate point data of the handwriting, it does not have line thickness information.

  FIG. 4 is a diagram illustrating an example of an edge image generated from a thin handwriting and a thick handwriting. As shown in FIG. 4, when handwriting with a narrow line width and thick handwriting are converted into edge images, the edges are along the outline of the character, and therefore the handwriting with different line widths has greatly different edge positions. Therefore, as shown in FIG. 4, when a handwriting image is generated from the coordinate point data, the search accuracy is reduced when the line width of the generated handwriting and the line width of the handwriting in the document are significantly different.

  In order to absorb the difference in line width, there is a technique for thinning a handwritten image (see, for example, JP 2009-53826 A). In this technique, the width of the handwriting image is reduced in an image processing manner so that the line width becomes 1.

  However, it is known that a general thinning technique cannot restore a handwriting line with high accuracy, and a lot of noise and inaccurate patterns are generated. FIG. 5 is a diagram illustrating a difference in the thinned image depending on the thickness of the character. As shown in FIG. 5, in an example in which handwriting having different line widths is thinned, a fine structure such as a part of a “brush” character is broken by a thick line.

  Therefore, it is difficult to search for handwriting written relatively small. However, if the handwriting image is generated with the same line width, since the crushing of a part of the character is similar, it is possible to detect even a handwriting image in which the character is crushed small.

  That is, if the line width of the handwriting is different, the fine structure collapses differently. Therefore, even if the handwriting having the same width is converted by thinning, it is difficult to detect the handwriting having a different line width.

  As described above, in order to realize the function of finding and deleting the handwritten handwriting from the document image, even if trying to find the corresponding handwriting in the document image using the coordinate data acquired using the electronic pen 10, Since the line width of the handwriting is not known, the search accuracy is lowered.

  If the type of the electronic pen 10 used for writing is known in advance, the line width of the handwriting is known to some extent. Even if the type is known, the line width changes depending on the amount of force applied when writing. Further, the tip of the electronic pen 10 is often detachable in order to replenish ink and change the color and thickness of the pen, and the thickness of the line when writing is usually unknown.

  Although it is possible in principle to let the user specify the line width, such a complicated operation impairs the usability for the user, so the normal electronic pen 10 specifies the pen thickness before writing. The function is not equipped. Therefore, when using the commercially available electronic pen 10, the line width of the handwriting is often unknown.

  Therefore, even if the line width of the handwriting is unknown, a function for finding the handwritten handwritten using the coordinate data acquired using the electronic pen 10 from the scanned document image and deleting the handwritten hand is required.

  Therefore, in the embodiment shown below, a plurality of handwriting line widths are prepared, a handwriting image most similar to the handwriting part in the document image is selected using the handwriting image of each line width, and the selected handwriting image is erased. Use. Hereinafter, embodiments will be described with reference to the accompanying drawings.

[Example]
<Hardware>
FIG. 6 is a block diagram illustrating an example of hardware of the image processing apparatus 100 according to the embodiment. In the example illustrated in FIG. 6, the image processing apparatus 100 is connected to the scanner device 20, the display device 30, and the electronic pen 10.

  The electronic pen 10 is a device that can write on paper or the like and can simultaneously acquire writing coordinate data. The electronic pen 10 transmits coordinate data acquired by wire or wireless to the image processing apparatus 100. The pen tip of the electronic pen 10 can be changed. As the electronic pen 10, for example, a commercially available product such as an ultrasonic electronic pen, anotopen, and a pen tablet can be applied.

  The ultrasonic electronic pen transmits ultrasonic and infrared pulses from the pen tip at a constant interval (about 100 times / second). Using the time difference between the two ultrasonic receivers, the coordinates of the pen tip are measured by the principle of triangulation.

  Anotopen uses special paper on which minute dots are printed. A camera for photographing the paper surface is attached to the pen tip, and the coordinates of the pen tip are recognized using a dot pattern photographed during writing.

  Pen tablets use a tablet that transmits and detects electromagnetic fields as an underlay. The nib usually has a built-in coil, and when written, the tablet can detect the coordinates of the nib by electromagnetic induction. A pole pen or the like can be attached to the pen tip, and both writing on paper and coordinate data acquisition are possible.

  The scanner device 20 scans an image on a paper surface and reads pixel values one by one with respect to x and y coordinates arranged two-dimensionally. When the pixel value to be read is a color image, the pixel value is represented by a set of three colors of R (red), G (green), and B (blue).

  When the image to be read is a gray image, the pixel value often represents a brightness value. The color pixel values of the three colors RGB can be converted into lightness values using, for example, a lightness value = 0.299 × R + 0.587 × G + 0.114 × B. The scanner device 20 transmits the read document image to the image processing device 100.

  The image processing apparatus 100 selects a handwriting image most similar to the handwriting portion in the document image using the handwriting image of each line width generated from the coordinate data, and erases the handwriting portion using the selected handwriting image. The image processing apparatus 100 may control to display a document image from which the handwriting portion has been deleted.

  The display device 30 is, for example, an LCD (Liquid Crystal Display) or the like, and displays the document image after erasing the handwritten portion acquired from the image processing device 100.

  The image processing apparatus 100 includes a control unit 101, a main storage unit 102, an auxiliary storage unit 103, a communication unit 104, and a drive device 105 as hardware. Each unit is connected via a bus so that data can be transmitted / received to / from each other.

  The control unit 101 is a CPU (Central Processing Unit) that performs control of each device, calculation of data, and processing in a computer. The control unit 101 is an arithmetic device that executes a program stored in the main storage unit 102 or the auxiliary storage unit 103. The control unit 101 receives data from the input device or the storage device, calculates, processes the output device, the storage device, and the like. Output to the device.

  The main storage unit 102 is, for example, a ROM (Read Only Memory) or a RAM (Random Access Memory). The main storage unit 102 is a storage device that stores or temporarily stores programs and data such as an OS and application software that are basic software executed by the control unit 101.

  The auxiliary storage unit 103 is, for example, an HDD (Hard Disk Drive) or the like, and is a storage device that stores data related to application software. The auxiliary storage unit 103 stores, for example, a document image acquired from the scanner device 20.

  The communication unit 104 performs data communication with peripheral devices by wire or wireless. The communication unit 104 acquires an image including characters via a network, for example, and stores it in the auxiliary storage unit 103.

  The drive device 105 reads a predetermined program from the recording medium 106, for example, a flexible disk or a CD (Compact Disc), and installs it in the storage device.

  In addition, a predetermined program is stored in the recording medium 106, and the program stored in the recording medium 106 is installed in the image processing apparatus 100 via the drive device 105. The installed predetermined program can be executed by the image processing apparatus 100.

  Note that the image processing apparatus 100 has the scanner device 20 and the display device 30 configured separately, but may include either one or both.

<Function>
FIG. 7 is a block diagram illustrating an example of functions of the image processing apparatus 100 according to the embodiment. In the example illustrated in FIG. 7, the image processing apparatus 100 includes an image acquisition unit 201, a handwriting acquisition unit 202, a generation unit 203, a search unit 204, a selection unit 205, and an erasure unit 206. Each unit illustrated in FIG. 7 functions, for example, when the control unit 101 loads and executes a writing erasure processing program stored in the auxiliary storage unit 103 in the main storage unit 102. Each unit illustrated in FIG. 7 can be realized by, for example, the control unit 101 and the main storage unit 102 as a work memory.

  The image acquisition unit 201 acquires a document image of a document including a handwriting by the electronic pen 10 from the scanner device 20. The image acquisition unit 201 outputs the acquired document image to the search unit 204 and the deletion unit 206.

  The handwriting acquisition unit 202 acquires handwriting coordinate data from the electronic pen 10 from the electronic pen 10. The handwriting acquisition unit 202 outputs the acquired coordinate data to the generation unit 203.

  The generation unit 203 generates a plurality of handwriting images based on the acquired coordinate data using a plurality of different line widths. The generation unit 203 outputs the generated plurality of handwriting images to the search unit 204.

  For each generated handwriting image, the searching unit 204 searches the document image for an image of a handwriting portion similar to the handwriting image. The search unit 204 performs a search using, for example, a generalized Hough transform. Further, the search unit 204 may use a handwriting image as a template and search using a pattern matching technique in the document image. The search unit 204 outputs the search result to the selection unit 205.

  The selection unit 205 selects a handwriting image that is most similar to the image of the handwriting part in the document image. For example, the selection unit 205 selects a handwriting image having the highest score using a score (score) representing the degree of coincidence calculated for each handwriting image. The selecting unit 205 outputs the selected handwriting image and the search result to the erasing unit 206.

  The eraser 206 erases the image of the handwriting part from the document image using the selected handwriting image.

  Next, an example in which the handwriting erasing process is performed using each unit in the embodiment will be described. FIG. 8 is a diagram for explaining the flow of handwriting erasing processing in the embodiment. The example shown in FIG. 8 is different from the example shown in FIG. 1 in that the generation unit 203 generates a plurality of handwriting images a207. The plurality of handwriting images a207 are handwriting images generated with different line widths.

  Since the search means 204 performs a search for each handwriting image, the search results are obtained by the number of handwriting images. The selection means 205 selects the handwriting image with the highest score among the search results. That is, the selection unit 205 selects a handwriting image having a line width most similar to the actual handwriting part.

  The erasing unit 206 can obtain the position and line width of the handwriting portion in the document image. By using these to erase the handwriting portion from the document image, the document image a204 of the original material can be generated. Next, each process related to the handwriting erasure process will be specifically described.

(Coordinate data acquisition process)
The electronic pen 10 may be a commercially available product such as an ultrasonic electronic pen or a form in which a paper document is placed on a tablet device and a ballpoint pen tip is attached to the tip of a stylus pen. For example, the electronic pen 10 may have a function of acquiring handwriting coordinate values at regular intervals (for example, at intervals of 0.01 seconds) and outputting an array of pairs of x coordinate values and y coordinate values.

  Specifically, in the coordinate data, a point from a point at which writing with the electronic pen 10 is started (pen down) to a point at which the electronic pen 10 is raised (pen up) represents one stroke of characters. The coordinate data is represented by an array in which such coordinate values are arranged in time order.

  FIG. 9 is a diagram illustrating an example of the data structure of the coordinate data. In the example shown in FIG. 9, the x coordinate, the y coordinate, and the writing state are recorded for each coordinate point. The x-coordinate and the y-coordinate are represented by 2-byte integer values. The writing state takes any value from 0 to 3.

  FIG. 10 is a diagram for explaining the meaning of the writing state. As shown in FIG. 10, “0” indicates a state in which the pen is in the air and is not written. “1” indicates the start point of the stroke. “2” indicates a coordinate point that is being written and is neither a start point nor an end point. “3” indicates the end point of the stroke. The handwriting acquisition unit 202 acquires coordinate data as illustrated in FIG. 9 from the electronic pen 10.

(Handwriting image generation processing)
The generation unit 203 generates a handwriting image that can be used by the search unit 204 described later, using the coordinate data as an input. Since the coordinate data is data in which coordinate points are discretely arranged (see, for example, FIG. 9), a state where coordinate points are connected by line segments represents an actual handwriting.

  FIG. 11 is a diagram illustrating an example of a state in which coordinate points are connected by line segments. As shown in FIG. 11, the generation unit 203 first connects the coordinate points included in the coordinate data with line segments. Next, the generation unit 203 generates a handwriting image by filling pixels between coordinate points with, for example, black pixels. A specific procedure will be described below.

  The generation unit 203 generates a handwriting image with a plurality of line widths. Here, a procedure for generating a handwriting image for one line width w is shown. First, the generation unit 203 fills pixels that overlap the line segments between the coordinate points with the line width w = 1. FIG. 12 is a diagram illustrating an example of a state in which a space between coordinate points is filled with one pixel. As shown in FIG. 12, pixels on a line segment connecting coordinate points are filled with, for example, black pixels. The color of the pixel to be filled is not limited to black, but may be any handwriting color.

  Next, the generation unit 203 draws an arrow having a set line width w in a direction orthogonal to the line segment, and connects the tips of the arrows with dotted lines. FIG. 13 is a diagram illustrating an example of a state in which the handwriting boundary is determined. As shown in FIG. 13, the area surrounded by the arrow and the dotted line is a pixel constituting the handwriting.

  Next, the generation unit 203 fills a portion where the enclosed range exceeds half of the pixels with black pixels. FIG. 14 is a diagram illustrating an example of a state where pixels are filled with a line width w. As shown in FIG. 14, the generation unit 203 can generate a handwriting image with a plurality of set line widths.

  Moreover, what is necessary is just to set a some value within the range of the line width which can be assumed for the line width of a handwriting. The value of the line width can be set arbitrarily. For example, assuming that the line thickness is in the range of 0.2 mm to 2 mm and the resolution of the scanned image is 200 dpi, 0.2 mm corresponds to about 1.57 pixels. 2 mm corresponds to 15.7 pixels.

  The line width is converted to an integer by allowing a slight error, and the generation unit 203 may generate a handwriting image having a width of 1 to 16 pixels, for example. The generation means 203 may thin out the line width of the generated handwriting image to increase the calculation efficiency, and may generate the handwriting image only with, for example, six types of line widths of 1, 4, 7, 10, 13, and 16.

  In addition, when the scanning resolution can be acquired from the scanner device 20, the generation unit 203 may change the range of the line width and the interval of the line width to be used according to the resolution of the document image. Even with the same line width in a paper document, the line width converted to a pixel value on the image is proportional to the resolution. For example, a line that is 1 pixel wide in a 200 dpi image is the same width on paper as 2 pixels wide in a 400 dpi image.

  Therefore, it is desirable that the range of the line width is enlarged or reduced almost in proportion to the resolution. On the other hand, since the degree of coincidence in the image search to be described later depends on the difference in pixel values, it is considered desirable to use a constant value for the interval of the line width used regardless of the resolution. Based on such a criterion, the generation unit 203 changes the line width setting according to the resolution.

  First, the upper limit of the line width in the case of 200 dpi is 16 pixels, and 16 pixels is used as a reference. Assuming that the resolution of the image is D, the upper limit value of the line width is rounded to the value obtained by D × 16/200. For example, if the resolution is 300 dpi, the upper limit of the line width is 300 × 16/200 = 24.

  Assuming that the minimum value of the line width is always 1 and the interval is 3 pixels, the generation unit 203 obtains the values from the minimum value of 1 pixel until the upper limit value is exceeded at intervals of 3 pixels. For example, when the resolution is 300 dpi, the generation unit 203 may use nine types of line widths of 1, 4, 7, 10, 13, 16, 19, 22, and 25. Here, the interval between the line widths is set to a constant value of 3 pixels, but the generation unit 203 does not fix the interval, fixes the type of line width to be used to 6 types, and divides the minimum value to the maximum value into 6 equal parts An approximate integer value may be used. As described above, the generation unit 203 may set the number and / or value of line widths according to the resolution of the document image.

(Search process)
The search means 204 searches the scanned document image for a position that matches the handwriting image to be searched. Hereinafter, the handwriting image to be searched is also referred to as a search handwriting image.

  FIG. 15 is a diagram illustrating an example of a search operation. The retrieval unit 204 sets a reference point for the retrieved handwriting image and finds a pattern that matches the handwriting from the document image. The search means 204 outputs the position coordinates of the reference point, the score of coincidence, and the size. As shown in FIG. 15, the search unit 204 maps the x axis and the y axis to corresponding positions in the document image with the origin of the relative position of the search handwritten image as a reference point.

  In the example shown in FIG. 15, the handwriting added to the document image may overlap with characters and ruled lines, and thus a technique capable of searching even if there is an overlap with other images is necessary. Therefore, for example, a generalized Hough transform can be used.

  In the generalized Hough transform, the search can be performed even when the size of the handwriting to be searched in the document image has changed or rotated. However, in the embodiment, it is assumed that there is no change in size and direction in order to simplify the description. When there is a possibility of size change or rotation, the search unit 204 may output not only the position coordinates of the reference point but also the size and rotation angle.

  Here, in the case of the method using the generalized Hough transform, the search means 204 votes the score (score) at the position of the reference point based on the edge direction extracted from the search handwriting image, and the sum of the voted values is the highest. Let the position be the search position. The score of the search result is always obtained, and in the embodiment, one search result is obtained for one search handwriting image.

  FIG. 16 is a diagram illustrating an example of a search result for each line width. As shown in FIG. 16, the search result holds a score, position coordinates, and size for each line width. The line width indicates a result when, for example, six types of line widths are used.

  Since the line width is a small value, it is stored in 1 byte, for example. The score assumes a value of 0 to 100, with 100 points when the score is perfect (completely matched) and 0 when the score does not match at all. The score can be expressed by, for example, 1 byte, but considering the case where the range of the score value is changed, a margin is assumed to be 2 bytes.

  The position coordinates indicate where the reference point is detected in the document image. Since the position coordinates are a set of x coordinates and y coordinates, an area of 2 bytes × 2 is used. The size is expressed by the length corresponding to the longer one (long side) of the search handwriting image in the document image. The search means 204 outputs the final search result as shown in FIG.

(Selection process)
The selection unit 205 selects a result having the highest search result score from a plurality of search handwriting images generated with a plurality of line widths. As described above, since the search unit 204 outputs one search result for one handwriting image, when a handwriting image is generated with a plurality of line widths, a plurality of search results (coordinate values of reference points) are obtained. can get. Of these, only one result is used to erase the handwritten image. The selection unit 205 selects a result having the maximum score from a plurality of results. The selecting unit 205 outputs the selected handwriting image and the search result to the erasing unit 206.

  When the example shown in FIG. 16 is used for the selection unit 205, the best result is obtained when the line width w = 7. Since w = 16 has a low score and the position coordinates are completely different, it can be seen that another place which is not correct is detected. For line widths other than w = 1 and 16, the correct handwriting can be detected, but the position coordinates and size are shifted little by little, which may affect the accuracy when the handwriting is erased later. .

(Erase processing)
The erasing unit 206 erases the handwriting image from the scanned document image. Since the erasing unit 206 already knows the position coordinates of the handwriting image in the document image, the pixel belonging to the handwriting to be erased can be found by overlaying the search image at that position. Therefore, the erasing unit 206 can erase the handwriting image by painting the pixels belonging to the handwriting with the background color. However, in the case of a color image, the background is not necessarily white, so the background color to be filled may be selected.

  If the document image is a black and white image and the background is white, and the additionally written handwriting is written only in a background area such as a margin, the erasing unit 206 only needs to fill the pixels of the handwriting with white. However, when writing is performed with overlapping printed characters or ruled lines, an unnatural image may be generated simply by painting the pixels.

  FIG. 17 is a diagram illustrating an example of simply painting with a background color. FIG. 17A shows a part of an image before additional recording. It is assumed that the black pixel indicates a part of the character. FIG. 17B shows the position of the image after additional recording. When the additionally written handwriting is written on the character, a portion where the character image and the handwriting overlap is generated as shown in FIG.

  FIG. 17C shows a part of an image in which a postscript portion is filled with a background color. When the pixels of the handwriting are filled with the background color, as shown in FIG. 17C, the characters are interrupted in the middle. However, the information for determining whether the pixel overwritten with the handwriting is the character pixel or the background is lost. Therefore, the image shown in FIG. 17C is not necessarily an inappropriate image, but in the example shown in FIG. 17C, it is more appropriate to generate an image as shown in FIG. it is conceivable that. It is also necessary to consider the case where the background is not limited to white, such as a color image.

  The erasing unit 206 in the embodiment performs processing for appropriately filling the background color by the following method. The erasing unit 206 first obtains the color of the non-handwriting pixel closest to the pixel for each image belonging to the handwriting.

  FIG. 18 is a diagram illustrating the filling process according to the embodiment. FIG. 18A shows the determination result of the fill color. In the example shown in FIG. 18A, a white pixel is indicated by A and a black pixel is indicated by B. Subsequently, the erasing unit 206 fills with the pixel color determined for each pixel. FIG. 18B is a diagram illustrating a part of an image after painting. In the example shown in FIG. 18B, the pixels B filled with black for easy understanding are shown in a grid-like net pattern.

  The erasing unit 206 obtains the color of the pixel other than the handwriting closest to the pixel as the color for filling each pixel. FIG. 19 is a diagram illustrating the fill color determination. In the example illustrated in FIG. 19, for example, the pixel C other than the handwriting is closest to a pixel at a position moved by two pixels in the upward direction. In this case, the distance is 2.

  In the case of the pixel D, when the pixel advances by 3 pixels in the upward direction, a white pixel as a background is found and becomes the closest pixel. On the other hand, the distance from the pixel D to the character pixel advances from one to the left and then advances upward by three pixels, so the distance is 4. As a result, the pixel closest to the pixel D becomes a background pixel having a distance of 3. When the fill determination is performed using the above method, the result shown in FIG.

  Further, the erasing unit 206 may perform the erasing process using a handwriting image whose line width is larger than the selected handwriting image by a predetermined value. This is because a handwriting image having a line width slightly smaller than the line width of the actual handwriting may be selected as the optimum, and this prevents the outer edge of the handwriting from remaining after erasing with the handwriting image. It is appropriate that the erasing unit 206 uses a handwriting image having a line width that is equal to or slightly larger than the actual line width of the handwriting. The predetermined value may be set to an appropriate value through experiments or the like in advance. Note that the erasing unit 206 may cause the generation unit 203 to generate a handwriting image whose line width is larger by a predetermined value than the selected handwriting image.

  The image from which the handwriting has been erased is displayed on the display device 30 or stored in the main storage unit 102 or the auxiliary storage unit 103.

  According to the above configuration and processing, the handwriting portion of the document image can be appropriately erased using the handwriting image generated from the coordinate data by the electronic pen.

<Operation>
Next, the flow of handwriting erasure processing in the embodiment will be described. FIG. 20 is a flowchart illustrating an example of handwriting erasing processing in the embodiment. In step S <b> 101 shown in FIG. 20, the user writes on the material using the electronic pen 10. At this time, the electronic pen 10 acquires handwriting coordinate data and transmits it to the image processing apparatus 100. The handwriting acquisition unit 202 of the image processing apparatus 100 acquires coordinate data from the electronic pen 10.

  In step S <b> 102, the scanner device 20 scans the material written with the electronic pen 10 to generate a document image. The generated document image is transmitted to the image processing apparatus 100. The image acquisition unit 201 of the image processing apparatus 100 acquires a document image.

  In step S103, the generation unit 203 generates a plurality of handwriting images using a plurality of different line widths based on the coordinate data. The handwriting image generation process will be described later with reference to FIG.

  In step S104, the retrieval unit 204 retrieves a similar partial image from the document image for each generated handwriting image, and calculates a score.

  In step S <b> 105, the selection unit 205 selects a handwritten image having a line width with the highest score from the search results.

  In step S106, the erasing unit 206 erases the handwriting portion from the document image using the handwriting image having the selected line width. The process of erasing the handwriting and filling the pixels with the background color will be described later with reference to FIG.

  With the above processing, the handwriting portion can be appropriately erased from the document image using the coordinate data acquired by the electronic pen 10.

  FIG. 21 is a flowchart illustrating an example of processing for generating a plurality of handwriting images. In step S201 shown in FIG. 21, the generation unit 203 first sets N types of line width values in an array W [i] (i = 1,..., N). Further, the generation unit 203 sets 1 to the variable i.

  In step S202, the generation unit 203 sets a value of W [i] for the line width w. In step S203, the generation unit 203 sets the stroke start point and the adjacent point as the start point and end point of each line segment.

  In step S204, the generation unit 203 sets the line width to 1 and fills pixels passing through the line segment (see, for example, FIG. 12). In step S205, the generation unit 203 determines whether or not the line width w = 1. If w = 1 (step S205—YES), the process proceeds to step S207, and if w = 1 is not satisfied (step S205—NO), the process proceeds to step S206.

  In step S206, the generation unit 203 fills the line segment within the range of the line width w (see, for example, FIGS. 13 and 14).

  In step S207, the generation unit 203 determines whether or not the handwriting end point has been reached. If the end point has been reached (step S207—YES), the process proceeds to step S209. If the end point has not been delivered (step S207—NO), the process proceeds to step S208.

  In step S208, the generation unit 203 sets the end point of the line segment as the next start point, and further sets the adjacent point as the end point of the line segment.

  In step S209, the generation unit 203 determines whether or not the variable i = N. If i = N (step S209—YES), this process ends. If i = N (step S209—NO), the process proceeds to step S210.

  In step S210, the generation unit 203 increments the value of i by 1, and returns to step S202.

  Through the above processing, handwriting images having a plurality of line widths can be generated. Furthermore, in step S201, when the generation unit 203 acquires the resolution at the time of scanning of the scanner device 20, the generation unit 203 may change the value and number of line widths based on this resolution.

  FIG. 22 is a flowchart illustrating an example of pixel filling processing after handwriting erasure. Hereinafter, the erased handwriting pixel is also referred to as handwriting pixel. In step S301 shown in FIG. 22, the erasing means 206 sets the color of all the handwritten pixels to “undecided”.

  In step S302, the erasing unit 206 selects one “undecided” handwriting pixel and sets it to the current processing target image.

  In step S303, the erasing unit 206 obtains the color of the pixel closest to the current processing target image.

  In step S304, the erasing unit 206 determines whether there are a plurality of closest pixels. If there are a plurality of closest pixels (step S304—YES), the process proceeds to step S305, and if there are not a plurality of closest pixels (step S304—NO), the process proceeds to step S306.

  In step S305, the erasing unit 206 obtains the average color of the closest plurality of pixels. In step S306, the erasing unit 206 sets the color obtained in step S303 or S305 as the color of the current processing target image.

  In step S307, the erasure unit 206 determines whether or not there is an “indeterminate” pixel. If there is no “undetermined” pixel (step S307—YES), this process is terminated. If there is an “undetermined” pixel (step S307—NO), the process returns to step S302.

  By the above processing, even if some of the document image information is lost because the pixels of the original document are overwritten by handwriting superimposition, comparison is made by suppressing unnatural phenomena such as character breaks. A natural document image can be restored.

  As described above, according to the embodiment, the handwriting portion of the document image can be appropriately erased using the handwriting image generated from the coordinate data by the electronic pen. Further, according to the embodiment, if the resolution at the time of scanning from the scanner device 20 can be acquired, the setting of the line width can be changed according to the resolution.

  By erasing the handwriting part of the document image, the visibility of the document image is improved, and at the same time, the accuracy of character recognition processing for search is improved, and the utility value of the document image can be improved.

  In addition, by recording a program for realizing the image processing described in the above-described embodiment on a recording medium, the image processing in the embodiment can be performed by a computer. For example, it is possible to record the program on a recording medium and cause the computer to read the recording medium on which the program is recorded, thereby realizing the above-described image processing.

  The image processing apparatus 100 according to the embodiment is applicable to any processing apparatus having a processor, a memory, and an interface for acquiring data, such as a computer (Personal Computer), a tablet terminal, and a smartphone. In addition, the scanner device 20 may be mounted so that the image processing can be executed by transmitting coordinate data from the electronic pen 10 to the scanner device 20.

  The recording medium is a recording medium that records information optically, electrically, or magnetically, such as a CD-ROM, flexible disk, magneto-optical disk, etc., and information is electrically recorded such as ROM, flash memory, etc. Various types of recording media such as a semiconductor memory can be used. This recording medium does not include a carrier wave.

  Although the embodiments have been described in detail above, the invention is not limited to the specific embodiments, and various modifications and changes can be made within the scope described in the claims. It is also possible to combine all or a plurality of the components of the above-described embodiments.

In addition, the following additional remarks are disclosed regarding the above Example.
(Appendix 1)
Image acquisition means for acquiring a document image of a document including handwriting by an electronic pen;
Handwriting acquisition means for acquiring coordinate data of handwriting by the electronic pen;
Generating means for generating a plurality of handwriting images based on the coordinate data using a plurality of different line widths;
Search means for searching for an image of a handwriting part similar to the handwriting image for each generated handwriting image from the document image;
Selecting means for selecting a handwriting image most similar to the image of the handwriting part in the document image;
Using the selected handwriting image, erasing means for erasing the image of the handwriting part from the document image,
An image processing apparatus comprising:
(Appendix 2)
The generating means includes
The image processing apparatus according to appendix 1, wherein the number and / or value of line widths are set according to the resolution of the document image.
(Appendix 3)
The generating means includes
The image processing apparatus according to appendix 2, wherein an upper limit value of a line width is calculated from the resolution of the document image, and the number of line widths is set using the upper limit value.
(Appendix 4)
The erasing means includes
The image processing apparatus according to any one of appendices 1 to 3, wherein an erasing process is performed using a handwriting image whose line width is larger by a predetermined value than the selected handwriting image.
(Appendix 5)
The erasing means includes
The image processing apparatus according to any one of appendices 1 to 4, wherein for the erased pixel, the color other than the erased pixel is used to restore the color using the color of the closest pixel.
(Appendix 6)
Obtain a document image of a document including handwriting with an electronic pen,
Acquire coordinate data of handwriting by the electronic pen,
Using a plurality of different line widths to generate a plurality of handwriting images based on the coordinate data,
For each generated handwriting image, search for an image of a handwriting part similar to the handwriting image from the document image,
Select the handwriting image most similar to the image of the handwriting part in the document image,
An image processing method in which a computer executes a process of deleting an image of a handwriting portion from the document image using a selected handwriting image.
(Appendix 7)
Obtain a document image of a document including handwriting with an electronic pen,
Acquire coordinate data of handwriting by the electronic pen,
Using a plurality of different line widths to generate a plurality of handwriting images based on the coordinate data,
For each generated handwriting image, search for an image of a handwriting part similar to the handwriting image from the document image,
Select the handwriting image most similar to the image of the handwriting part in the document image,
A program that causes a computer to execute a process of erasing an image of a handwriting portion from the document image using a selected handwriting image.

DESCRIPTION OF SYMBOLS 10 Electronic pen 20 Scanner apparatus 100 Image processing apparatus 101 Control part 102 Main memory part 103 Auxiliary memory part 104 Communication part 105 Drive apparatus 201 Image acquisition means 202 Handwriting acquisition means 203 Generation means 204 Search means 205 Selection means 206 Erase means

Claims (6)

Image acquisition means for acquiring a document image of a document including handwriting by an electronic pen;
Handwriting acquisition means for acquiring coordinate data of handwriting by the electronic pen;
Generating means for generating a plurality of handwriting images based on the coordinate data using a plurality of different line widths;
Search means for searching for an image of a handwriting part similar to the handwriting image for each generated handwriting image from the document image;
Selecting means for selecting a handwriting image most similar to the image of the handwriting part in the document image;
Using the selected handwriting image, erasing means for erasing the image of the handwriting part from the document image,
An image processing apparatus comprising: The generating means includes
The image processing apparatus according to claim 1, wherein the number and / or value of line widths are set according to the resolution of the document image. The generating means includes
The image processing apparatus according to claim 2, wherein an upper limit value of a line width is calculated from the resolution of the document image, and the number of line widths is set using the upper limit value. The erasing means includes
4. The image processing apparatus according to claim 1, wherein the erasing process is performed using a handwriting image whose line width is larger by a predetermined value than the selected handwriting image. 5. Obtain a document image of a document including handwriting with an electronic pen,
Acquire coordinate data of handwriting by the electronic pen,
Using a plurality of different line widths to generate a plurality of handwriting images based on the coordinate data,
For each generated handwriting image, search for an image of a handwriting part similar to the handwriting image from the document image,
Select the handwriting image most similar to the image of the handwriting part in the document image,
An image processing method in which a computer executes a process of deleting an image of a handwriting portion from the document image using a selected handwriting image. Obtain a document image of a document including handwriting with an electronic pen,
Acquire coordinate data of handwriting by the electronic pen,
Using a plurality of different line widths to generate a plurality of handwriting images based on the coordinate data,
For each generated handwriting image, search for an image of a handwriting part similar to the handwriting image from the document image,
Select the handwriting image most similar to the image of the handwriting part in the document image,
A program that causes a computer to execute a process of erasing an image of a handwriting portion from the document image using a selected handwriting image.

Images