JP2008123456A - Image processing method, image processing device, image forming device, computer program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method, an image processing device, an image forming device, and a computer program capable of extracting an inserted image inserted in an original image more easily than ever before, and a recording medium in which the computer program is recorded. <P>SOLUTION: A document collation processing part 22 calculates a hash value based on an inputted original image, judges whether or not the original image is similar to the registration format stored beforehand based on the calculated hash value, and, if it is judged to be similar, judges whether or not an insertion is made into the original image, and extracts the images inserted into the original image. The document collation processing part 22 collates the extracted images inserted into the original image with the inserted images stored in association with the registration format judged to be similar to the original image, and extracts the inserted images similar to the images inserted into the original image. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing method, an image processing apparatus, and an image forming apparatus including the image processing apparatus, which collate an acquired document image with a registered image registered in advance and perform processing on the document image based on the collation result. The present invention relates to a computer program for realizing the image processing apparatus and a recording medium on which the computer program is recorded.

  As a process of reading an original with a scanner and determining whether the read original image is similar to a pre-registered format, a method of extracting a keyword from an image read with an OCR and performing pattern matching with the extracted keyword, or A method is proposed in which the document to be determined is limited to a form document with ruled lines, and pattern matching is performed based on information about ruled lines extracted from the read document.

  To register the format used for similarity determination, recognition processing such as line segment extraction, character frame extraction, character recognition, or frame extraction is performed on the input image input for registration, and the center of the frame data is determined from the recognition result. Information (for example, feature points) such as coordinates, center coordinates of character string frames, and connection frame information is extracted. Next, a feature amount (for example, a hash value) is calculated from the extracted information, and data necessary for table management (feature amount, model name, parameters used for calculating the feature amount, etc.) is created and a hash table is created. By registering with, format registration is performed.

When similarity determination of a document is performed, recognition processing is performed on an input document image, and information such as center coordinates of frame data, center coordinates of a character string frame, connection frame information, and the like (for example, feature points) from the recognition result. ). Then, a feature amount (for example, a hash value) for each information is calculated, a pre-registered hash table area is searched using the calculated feature amount, and voting is performed for each registered form name in the searched area. . These processes are repeated for each feature point of the input document image, and the similarity is calculated using the model with the largest histogram as the recognition result. When it is confirmed that the document image is similar to the registered format, an identifier is assigned to the document image and stored. Also, when it is determined which type of registered form the manuscript (input form) is, it is associated with the frame data from the plot position of the hash table, the image within the frame is cut out, and the registered form name and Using the associated file name, save only the cut out partial image. By performing such processing, an image (original image) filing device has been proposed that can automatically match an original image with a registered format and reduce the user's processing steps (patent). Reference 1).
Japanese Patent No. 3469345

  However, in the apparatus of Patent Document 1, it is pointed out that efficient file management or search can be performed by associating the file name with the registered form name and the attribute indicated by the frame data. However, if there is a large amount of registered data, it is necessary to search after confirming the file name associated with the registered form name or frame data, so to extract the written image written on the original image, There is a possibility that the processing required for the search becomes enormous and takes a long time. In addition, when a part of the document is dirty or missing, it is impossible to retrieve the original written image written on the document by searching the registered format.

  The present invention has been made in view of such circumstances, and determines whether or not an original image is similar to a registered image. If the original image is similar to a registered image, the original writing written on the original image is performed. An image is specified, and it is determined whether or not the specified document writing image is similar to a writing image associated with a registered image similar to the document image. If the document writing image is similar to the writing image, the writing image In order to realize an image processing method, an image processing apparatus, an image forming apparatus provided with the image processing apparatus, and the image processing apparatus, which can extract a written image written on a document more easily than in the past. It is an object of the present invention to provide a computer program and a recording medium on which the computer program is recorded.

  Another object of the present invention is to accept selection of a location where a written image of a registered image is written, and to determine whether or not the original image is written, it corresponds to the selected location of the original image. If it is determined whether or not the specified original writing image is similar to the registered image writing image, by excluding the writing image of the selected portion, a part of the original is soiled or missing. The present invention is to provide an image processing method, an image processing apparatus, and an image forming apparatus including the image processing apparatus that can extract a written image written on a document even if there is a problem.

  Another object of the present invention is to synthesize a registered image associated with the written image and the written image when it is determined that the original written image is similar to the written image, and output the combined registered image and written image. Accordingly, an object of the present invention is to provide an image processing method, an image processing apparatus, and an image forming apparatus including the image processing apparatus that can restore the original document that has been written.

  Another object of the present invention is to determine whether or not a document image has been written, to determine the presence or absence of a background based on the density value of each pixel of the document image. It is an object of the present invention to provide an image processing method, an image processing apparatus, and an image forming apparatus including the image processing apparatus that can accurately extract a written image written on a document by removing the image.

  Another object of the present invention is to provide an image processing apparatus capable of storing a written image written on a document in association with a registered image, and an image forming apparatus including the image processing apparatus.

  An image processing method according to the present invention is an image processing method for determining whether an acquired document image is similar to a registered image registered in advance, and extracting a written image written in the registered image based on the determination result. A plurality of write images written in one or a plurality of locations determined for each of the plurality of registered images are stored in association with the registered image, and it is determined whether the document image is similar to the registered image. If the document image is similar to the registered image, it is determined whether or not the document image has been written. If the document image has been written, the document writing image written in the document image is identified, and the identified document is identified. It is determined whether the written image is similar to a written image associated with a registered image similar to the original image. If the original written image is similar to the written image, the written image is extracted. It is characterized in.

  The image processing method according to the present invention accepts selection of a place where a written image of a registered image is written, and when determining whether or not writing is performed on the original image, corresponds to the selected place of the original image. When the portion is excluded and it is determined whether or not the specified original writing image is similar to the writing image of the registered image, the writing image at the selected portion is excluded.

  In the image processing method according to the present invention, when it is determined that the original written image is similar to the written image, the registered image associated with the written image and the written image are combined, and the combined registered image and written image are output. It is characterized by that.

  In the image processing method according to the present invention, when determining whether or not a document image has been written, the presence or absence of a background is determined based on the density value of each pixel of the document image. It is characterized by removing.

  An image processing apparatus according to the present invention is an image processing apparatus that determines whether or not an obtained document image is similar to a registered image registered in advance, and extracts a written image written in the registered image based on the determination result. A means for storing a plurality of write images written in one or a plurality of locations determined for each of the plurality of registered images in association with the registered image, and determining whether the document image is similar to the registered image. And means for determining whether or not the original image is written when the original image is similar to the registered image, and specifying the original written image written on the original image when the original is written Means for determining whether the specified document writing image is similar to a writing image associated with a registered image similar to the document image; and If you like, characterized in that it comprises means for extracting 該書 fitted image.

  The image processing apparatus according to the present invention includes: a receiving unit that receives a selection of a portion where a written image of a registered image is written; and a determination unit that determines whether or not a document image has been written. When determining whether to exclude a portion corresponding to the selected portion and whether the specified original writing image is similar to the writing image of the registered image, the writing image at the portion selected by the receiving unit is excluded. Means.

  When it is determined that the original writing image is similar to the writing image, the image processing apparatus according to the present invention combines a registered image associated with the writing image with the writing image, and a registered image combined by the means. And means for outputting a written image.

  The image processing apparatus according to the present invention, when determining whether or not the document image has been written, when determining whether there is a background based on the density value of each pixel of the document image, And a means for removing the ground.

  An image processing apparatus according to the present invention includes means for accepting registration of an original writing image written in an original image, and means for specifying a registered image similar to the original image when registration of the original writing image is accepted by the means. And means for specifying an original writing image written in the original image, and means for storing the original writing image specified by the means in association with a location determined for the registered image. And

  An image forming apparatus according to the present invention includes: the image processing apparatus according to any one of the above-described inventions; and an image forming unit that forms an output image based on an image processed by the image processing apparatus. And

  A computer program according to the present invention causes a computer to determine whether or not an acquired document image is similar to a registered image registered in advance, and to extract a written image written in the registered image based on the determination result In the computer program, the computer determines whether the original image is similar to the registered image, and the computer determines whether the original image is written when the original image is similar to the registered image. And means for specifying the original writing image written in the original image, if the computer is written, and the computer for a registered image in which the specified original writing image is similar to the original image. Means for determining whether or not the image is similar to a written image written in one or more predetermined locations, and a computer If the document written image is similar to the write image, characterized in that to function as a means for extracting the 該書 fitted image.

  A recording medium readable by a computer according to the present invention records the computer program according to the above-described invention.

  In the present invention, it is determined whether or not the acquired document image is similar to a registered image registered in advance. If it is determined that the document image is similar to the registered image, the acquired document image is written. It is determined whether or not. When writing is performed on the document image, the document writing image written on the document image is specified. It is determined whether or not the specified document writing image is similar to a writing image associated with a registered image similar to the document image. If the document writing image is similar to the writing image, the writing image is extracted. That is, the registered image similar to the document image is specified by collating the document image with the registered image. The registered image is associated with a written image written at a predetermined location, and the original image of the original image is written by collating the original written image written on the original image with the written image associated with the registered image. A written image similar to the image is extracted. As a result, the original written image written on the original is extracted.

  In the present invention, selection of a portion where a written image of a registered image is written is accepted. For example, when a registered image similar to the document image is identified by comparing the document image with the registered image, a predetermined portion of the registered image where the written image is written is displayed. Thus, selection of a required part by the user is accepted. The selected part is, for example, a part corresponding to a part where the original is dirty or a part including a missing part. When determining whether or not a document image has been written, the portion corresponding to the selected portion of the document image is excluded, and it is determined whether or not the specified document written image is similar to the registered image. If so, the written image at the selected location is excluded. As a result, even if there is a dirty or missing part in the document, it is possible to determine whether or not the document image is written by excluding that part, and to verify the document written image and the registered image written image, Even when a part of the document is missing or dirty, the original written image of the document can be extracted.

  Further, in the present invention, when the original written image of the original image is compared with the written image of the registered image and it is determined that the original written image is similar to the written image, the registered image associated with the written image The written image is synthesized and the synthesized registered image and written image are output. As a result, the original written image can be synthesized with the registered image similar to the original image, and the original original can be restored.

  Further, in the present invention, when determining whether or not a document image has been written, the presence or absence of a background is determined based on the density value of each pixel of the document image. Remove. For example, the maximum density value determined to be the background is set in advance as the first threshold value, and how many pixels the number of pixels is determined to be the background, that is, the background is determined. A minimum value of the number of pixels to be processed is determined in advance as a second threshold value. Based on the density value of each pixel of the original image, a density histogram is created, and the density classification below the first threshold determined to be the background is obtained by searching from the high density side of the created density histogram. , A density section having the number of pixels equal to or greater than the second threshold is extracted as a background. Background removal processing is performed using a density correction table corresponding to the density classification extracted as the background. As a result, it is possible to absorb the influence of the fading of the original paper, the influence of fluctuations in the reading conditions when the image data is read by the scanner.

  In the present invention, when registration of a document writing image written on a document image is received, a registered image similar to the document image is specified and a document writing image written on the document image is specified. By storing the original writing image specified in association with a predetermined location for the specified registered image, the original writing image written on the original is registered as a writing image.

  In the present invention, it is determined whether or not the document image is similar to the registered image. If the document image is similar to the registered image, the document writing image written in the document image is specified, and the specified document writing is specified. It is determined whether or not an image is similar to a written image associated with a registered image similar to the original image, and when the original written image is similar to the written image, the written image is extracted, so that The written image written on the original can be extracted easily.

  In the present invention, when receiving a selection of a portion where a written image of a registered image is written and determining whether or not writing is performed on a document image, a portion corresponding to the selected portion of the document image is selected. When determining whether or not the specified original writing image is similar to the writing image of the registered image, by excluding the writing image at the selected location, a part of the original is soiled or missing. Even in this case, it is possible to extract the written image written on the document.

  In the present invention, when it is determined that the original written image is similar to the written image, the registered image associated with the written image is combined with the written image, and the combined registered image and written image are output. The original document that has been written can be restored.

  In the present invention, when determining whether or not writing has been performed on the document image, the presence or absence of a background is determined based on the density value of each pixel of the document image, and if there is a background, the background is removed. Thus, it is possible to accurately extract the written image written on the original.

  In the present invention, the original writing image written on the original can be stored in association with the registered image.

Embodiment 1
Hereinafter, the present invention will be described with reference to the drawings illustrating embodiments. FIG. 1 is a block diagram illustrating a configuration of an image forming apparatus 100 including an image processing apparatus according to the present invention. The image forming apparatus 100 (for example, a digital color copier, a multi-function machine, a multi-function machine having a printer function, a fax function or an e-mail delivery function) includes a color image input device 1, a color image processing device 2 (image processing device), an image A color image output device 3 as a forming unit, an operation panel 4 for performing various operations, and the like are provided. Image data of RGB (R: red, G: green, B: blue) analog signals obtained by reading a document with the color image input device 1 is output to the color image processing device 2, and the color image processing device 2. Then, a predetermined process is performed and output to the color image output device 3 as a digital color signal of CMYK (C: cyan, M: magenta, Y: yellow, K: black).

  The color image input device 1 is, for example, a scanner including a CCD (Charged Coupled Device), reads a reflected light image from a document image as an RGB analog signal, and outputs the read RGB signal to the color image processing device 2. . The color image output device 3 is an image forming unit using an electrophotographic system or an inkjet system that outputs image data of a document image onto a recording sheet. The color image output device 3 may be a display device such as a display.

  The color image processing apparatus 2 includes a CPU, an application specific integrated circuit (ASIC), and the like.

  The A / D conversion unit 20 converts the RGB signal input from the color image input device 1 into, for example, a 10-bit digital signal, and outputs the converted RGB signal to the shading correction unit 21.

  The shading correction unit 21 performs correction processing for removing various distortions generated in the illumination system, imaging system, imaging system, and the like of the color image input apparatus 1 on the input RGB signal. Further, the shading correction unit 21 performs processing for converting the signal such as a density signal into a signal that can be handled easily by the image processing system employed in the color image processing apparatus 2, and outputs the corrected RGB signal (image) to the document matching processing unit 22. Output.

  The document matching processing unit 22 binarizes the input image, calculates a feature point (for example, the center of gravity) of the connected region specified based on the binary image, and a plurality of feature points from the calculated feature points Is selected, an invariant is obtained based on the selected feature point, and a feature amount (for example, a hash value) is calculated based on the invariant. The document collation processing unit 22 determines (collates) whether or not an image (original image) input based on the calculated feature amount is similar to a registered format (registered image) stored in advance, and determines that they are similar. In this case, it is determined whether or not the input image (original image) has been written, and the written image (original written image) is extracted.

  Further, the document collation processing unit 22 collates the extracted document writing image with the writing image stored in association with the registered format determined to be similar to the document image as a result of the collation, and writes similar to the document writing image. Extract images. The document collation processing unit 22 outputs processing results such as a collation result between a document image and a registered format, a determination result of the presence / absence of a document written image, and a written image extraction result associated with the registered format. In addition, the document collation processing unit 22 outputs the input RGB signal as it is to the subsequent input tone correction unit 23.

  The input tone correction unit 23 performs a process of adjusting the color balance on the input RGB signal (RGB reflectance signal), and performs an image quality adjustment process such as removal of background density or contrast, and the processed RGB signal The signal is output to the region separation processing unit 24.

  Based on the input RGB signal, the region separation processing unit 24 separates whether each pixel in the input image is a character region, a halftone dot region, or a photographic region. Based on the separation result, the region separation processing unit 24 generates a region identification signal indicating which region each pixel belongs to, a black generation and under color removal unit 26, a spatial filter processing unit 27, an output tone correction unit 28, a floor. Output to the key reproduction processing unit 29. The region separation processing unit 24 outputs the input RGB signal as it is to the subsequent color correction unit 25.

  The color correction unit 25 converts the input RGB signal into a CMY color space, performs color correction in accordance with the characteristics of the color image output device 3, and outputs the corrected CMY signal to the black generation and under color removal unit 26. To do. Specifically, the color correction unit 25 performs a process of removing color turbidity based on the spectral characteristics of CMY color materials including unnecessary absorption components in order to make color reproduction faithful.

  The black generation and under color removal unit 26 generates a K (black) signal based on the CMY signal input from the color correction unit 25 and generates a new CMY signal by subtracting the K signal from the input CMY signal. Then, the generated CMYK signal is output to the spatial filter processing unit 27.

  An example of processing in the black generation and under color removal unit 26 is shown. For example, in the process of generating black by skeleton black, the input / output characteristic of the skeleton curve is y = f (x), the input data is C, M, Y, and the output data is C ′, M ′. , Y ′, K ′, and the UCR (Under Color Removal) rate is α (0 <α <1), the data output by the black generation and under color removal processing is K ′ = f {min (C, M, Y)}, C ′ = C−αK ′, M ′ = M−αK ′, and Y ′ = Y−αK ′.

  The spatial filter processing unit 27 performs spatial filter processing on the CMYK signal input from the black generation and under color removal unit 26 using a digital filter based on the region identification signal. As a result, the spatial frequency characteristics of the image data are corrected, and blurring of the output image in the color image output device 3 or deterioration of graininess is prevented. For example, the spatial filter processing unit 27 performs sharp enhancement processing on the region separated into character regions by the region separation processing unit 24 to enhance the reproducibility of black characters or color characters, and emphasizes high frequency components. The spatial filter processing unit 27 performs low-pass filter processing for removing the input halftone component on the region separated into the halftone dot region by the region separation processing unit 24. The spatial filter processing unit 27 outputs the processed CMYK signal to the output tone correction unit 28.

  The output tone correction unit 28 performs an output tone correction process for converting the CMYK signal input from the spatial filter processing unit 27 into a halftone dot area ratio that is a characteristic value of the color image output device 3, and The CMYK signal after the tone correction processing is output to the gradation reproduction processing unit 29.

  The gradation reproduction processing unit 29 performs predetermined processing on the CMYK signal input from the output gradation correction unit 28 based on the region identification signal input from the region separation processing unit 24. For example, the tone reproduction processing unit 29 performs binarization processing or the like so as to be suitable for reproduction of high-frequency components in the color image output device 3 in order to improve the reproducibility of the area separated into character areas, particularly black characters or color characters. Multilevel processing is performed.

  In addition, the gradation reproduction processing unit 29 performs gradation reproduction processing so that the region separated by the halftone dot region in the region separation processing unit 24 can be finally separated into pixels and the respective gradations can be reproduced. (Generate halftone). Further, the gradation reproduction processing unit 29 performs binarization processing or multi-value quantization processing on the region separated into the photographic region by the region separation processing unit 24 so as to be suitable for gradation reproducibility in the color image output device 3. .

  The color image processing apparatus 2 temporarily stores the image data (CMYK signal) processed by the gradation reproduction processing unit 29 in a storage unit (not shown), and stores the image data in the storage unit at a predetermined timing for image formation. And the read image data is output to the color image output device 3. These controls are performed by a CPU (not shown), for example. Further, a CPU (not shown) in the color image processing apparatus 2 controls output processing (for example, filing, copying, electronic distribution, etc.) for a document image based on the processing result output from the document matching processing unit 22. .

  The operation panel 4 includes, for example, a display unit such as a liquid crystal display and a setting button. The color image input device 1, the color image processing device 2, and the color image output device 3 are based on information input from the operation panel 4. Operation is controlled. For example, a registration mode / search mode selection button is displayed on the operation panel 4, and the user can select the registration mode / search mode. By selecting the registration mode, the registration format (registered image) or the original writing image written on the original image can be registered (stored). In addition, by selecting the search mode, part of the document is missing or dirty, so when the original document is output again, or information such as characters, symbols, and charts written on the document (writing When it is necessary to change the written image and the written information is replaced, the original written image can be extracted based on the original image.

  FIG. 2 is a block diagram showing a configuration of the document matching processing unit 22. The document collation processing unit 22 includes a feature point calculation unit 220, a feature amount calculation unit 221, a voting processing unit 222, a similarity determination processing unit 223, a writing area extraction unit 224, a registration control unit 225, a compression / decompression processing unit 226, a search A processing unit 227, a synthesis processing unit 228, a control unit 229 for controlling the above-described units, a memory 50, and the like are provided.

  The feature point calculation unit 220 performs predetermined processing to be described later on the input image (original image), extracts a character string or ruled line connection region of the input image, and extracts feature points (for example, the center of gravity) of the connection region. ) Is calculated. More specifically, the feature point calculation unit 220 binarizes the input image, and the feature points of the connected area specified based on the binary image (for example, the binary image of each pixel constituting the connected area). (The value obtained by dividing the accumulated coordinate value by the number of pixels included in the connected region) is extracted (calculated), and the extracted feature points are output to the feature amount calculation unit 221.

  FIG. 3 is a block diagram illustrating a configuration of the feature point calculation unit 220. The feature point calculation unit 220 includes an achromatic processing unit 2200, a resolution conversion unit 2201, a filter processing unit 2202, a binarization processing unit 2203, a centroid calculation unit 2204, and the like.

If the input image is a color image, the achromatic processing unit 2200 achromatizes the color image, converts it to a luminance signal or a brightness signal, and outputs the converted image to the resolution conversion unit 2201. For example, the luminance signal Yj can be expressed as Yj = 0.30 × Rj + 0.59 × Gj + 0.11 × Bj, where Rj, Gj, and Bj are the color components of each pixel RGB. Note that the RGB signal is not limited to the above formula, and can be converted into a CIE1976L ^* a ^* b ^* signal.

  Even when the input image is optically scaled by the color image input apparatus 1, the resolution conversion unit 2201 scales the input image again so that the predetermined resolution is obtained, and the scaled image is scaled. The image is output to the filter processing unit 2202. As a result, even when scaling processing is performed in the color image input apparatus 1 and the resolution changes, feature points can be extracted without being affected by the change, and the originals can be classified with high accuracy. . In particular, in the case of a reduced character or the like, when binarization processing is performed and a connected region is specified, the character is crushed and the originally separated regions are connected and the calculated center of gravity is shifted. Fear can be prevented. Also, the resolution conversion unit 2201 converts the resolution to a resolution smaller than the resolution read at the same magnification by the color image input apparatus 1. For example, an image read at 600 dpi (dot per inch) in the color image input device 1 is converted to 300 dpi. Thereby, the processing amount in the latter stage can be reduced.

  The filter processing unit 2202 corrects the spatial frequency characteristics of the input image (for example, image enhancement processing and smoothing processing), and outputs the corrected image to the binarization processing unit 2203. Since the spatial frequency characteristics of the color image input device 1 are different for each model, the filter processing unit 2202 corrects the different spatial frequency characteristics to the required characteristics. An image (for example, an image signal) output from the color image input apparatus 1 includes optical parts such as a lens or a mirror, aperture aperture of a light receiving surface of a CCD, transfer efficiency, afterimage, integration effect by physical scanning, and scanning. Deterioration such as image blur occurs due to unevenness. The filter processing unit 2202 restores deterioration such as blurring that has occurred in an image by performing enhancement processing such as a boundary or an edge. Further, the filter processing unit 2202 performs a smoothing process for suppressing high-frequency components that are unnecessary for the feature point extraction process to be processed later. As a result, feature points can be extracted with high accuracy, and as a result, the similarity of images can be determined with high accuracy. Note that the filter coefficient used in the filter processing unit 2202 can be appropriately set according to the model or characteristics of the color image input apparatus 1 used.

  FIG. 4 is an explanatory diagram illustrating an example of filter coefficients of the filter processing unit 2202. As shown in the figure, the spatial filter is, for example, a mixed filter having a size of 7 × 7 (7 rows, 7 columns) and performing enhancement processing and smoothing processing. The pixels of the input image are scanned, and arithmetic processing using a spatial filter is performed on all the pixels. Note that the size of the spatial filter is not limited to 7 × 7, and may be 3 × 3, 5 × 5, or the like. The numerical value of the filter coefficient is an example, and is not limited thereto, and can be set as appropriate according to the model or characteristics of the color image input apparatus 1 used.

  The binarization processing unit 2203 binarizes the image by comparing the luminance value (luminance signal) or brightness value (brightness signal) of the input image with a threshold value, and the binarized binary image 2204.

  Based on the binarization information (for example, represented by “1” and “0”) of each pixel of the binary image input from the binarization processing unit 2203, the centroid calculation unit 2204 Labeling (labeling processing) is performed, a connected region in which pixels with the same label are connected is specified, the center of gravity of the specified connected region is extracted as a feature point, and the extracted feature point is sent to the feature amount calculation unit 221. Output. The feature points can be represented by coordinates (x ′, y ′) in the binary image (original image).

  FIG. 5 is an explanatory diagram illustrating an example of feature points of a connected region. In the figure, the specified connected area is the letter “A”, and is specified as a set of pixels with the same label. The feature point (centroid) of the character “A” is a position (x ′ coordinate, y ′ coordinate) indicated by a black circle in the drawing.

  FIG. 6 is an explanatory diagram showing an example of a feature point extraction result for a character string. In the case of a character string composed of a plurality of characters, a plurality of feature points having different coordinates depending on the character type are extracted.

  The feature amount calculation unit 221 uses the feature points input from the feature point calculation unit 220 to obtain invariants for processing such as rotation, enlargement, and reduction, and based on the invariants, feature amounts (for example, hash values) Is calculated. More specifically, the feature quantity calculation unit 221 uses each feature point (that is, the coordinate value of the center of gravity of the connected area) input from the feature point calculation unit 220 as a feature point, for example, a distance from the feature point Four other feature points in the vicinity of small are extracted.

  FIG. 7 is an explanatory diagram showing a feature point of interest and surrounding feature points. As shown in the drawing, for example, four feature points surrounded by a closed curve S1 are extracted from the feature point P1 (the feature point P2 is also extracted as one feature point for the feature point P1). ing). Further, for example, four feature points surrounded by the closed curve S2 are extracted from the target feature point P2 (the target feature point P1 is also extracted as one feature point for the target feature point P2).

  The feature amount calculation unit 221 calculates three invariants by selecting three feature points from the extracted four feature points. Note that the number of feature points to be selected is not limited to three, and four, five, etc. feature points may be selected. The number of feature points to be selected differs depending on the type of invariant to be obtained. For example, an invariant obtained from three points is a similar invariant.

  FIG. 8 is an explanatory diagram showing an example of calculating an invariant using the feature point of interest P1, and FIG. 9 is an explanatory diagram showing an example of calculating an invariant using the feature point of interest P2. As shown in FIG. 8, three feature points are selected from four feature points around the target feature point P1, and each of the three invariants is set to H1j (j = 1, 2, 3). The invariant H1j is calculated by an expression represented by H1j = (A1j / B1j) × C / D. Here, A1j and B1j each indicate a distance between feature points, C is a normalization constant, and D is a predetermined constant. Thereby, for example, even when the document is rotated, moved, or tilted, the invariant H1j does not change, and the similarity of images can be determined with high accuracy, and similar documents can be determined with high accuracy. Can do.

  Similarly, as shown in FIG. 9, three feature points are selected from the four feature points around the target feature point P2, and each of the three invariants is set to H2j (j = 1, 2, 3). The invariant H2j is calculated by an expression represented by H2j = (A2j / B2j) × C / D. Here, A2j and B2j each indicate a distance between feature points, C is a normalization constant, and D is a predetermined constant. Similarly, invariants can be calculated for other attention feature points.

The feature amount calculation unit 221 calculates a hash value (feature amount) Hi based on the invariant calculated by each feature point of interest. The hash value Hi of the feature point Pi of interest is represented by Hi = (Hi1 × 10 ² + Hi2 × 10 ¹ + Hi3 × 10 ⁰ ) / E. Here, E is a constant determined by how much the remainder is set. For example, when “10” is set, the remainder is “0” to “9”, and the hash value calculated by this can be taken. It becomes a range. I is a natural number. The hash value as the feature amount is an example, and the hash value is not limited to this, and other hash functions can be used. In the above, an example in which four points are extracted as other feature points in the vicinity is shown, but the present invention is not limited to four points. For example, six points may be extracted. In this case, 5 points are extracted from 6 feature points, and for each of the 6 methods of extracting 5 points, 3 points are extracted from 5 points to obtain an invariant, and a hash value is calculated. good. Thus, one hash value can be calculated from one feature point, and a plurality of hash values can be calculated from one feature point.

  The feature amount calculation unit 221 stores an index representing a feature point of the document image, a hash value calculated based on the feature point, and coordinates of the feature point on the document image as a feature point coordinate table of the document image. 50.

  In addition, when the registration mode of the registration format (registered image) is selected on the operation panel 4, the feature amount calculation unit 221 includes an index representing the feature point of the registered image, a hash value calculated based on the feature point, and The coordinates of the feature point on the registered image are output to a registration control unit 225 described later.

  FIG. 10 is an explanatory diagram showing the structure of the feature point coordinate table 501 of the document image. The feature point coordinate table 501 of the document image includes columns of an index representing the feature point of the document image, a hash value calculated based on the feature point, and coordinates (x ′ coordinate, y ′ coordinate) of the feature point on the document image. It is comprised by.

  As shown in FIG. 10, for example, the hash value calculated based on the feature point whose index representing the feature point of the document image is p1 is H1, and the coordinates of the feature point are (x11 ′, y11 ′). Similarly, the hash value calculated based on the feature point whose index representing the feature point of the original image is p2 is H8, and the coordinates of the feature point are (x32 ′, y32 ′).

  FIG. 11 is an explanatory diagram showing the structure of the hash table 502 in the registration format. As shown in FIG. 11A, the structure of the hash table 502 is composed of columns of hash values and indexes representing originals (registered formats). More specifically, a point index indicating the position in the document and an invariant (both not shown) are registered corresponding to the index indicating the document. The hash table 502 is stored in the memory 50. In addition, a registration format (image to be collated / document image) is stored in the memory 50 in advance in order to determine the similarity of images. As shown in FIG. 11B, when the hash values are equal (H1 = H5), two entries of the hash table 502 can be combined into one.

  FIG. 12 is an explanatory diagram showing the structure of the feature point coordinate table 503 of the registration format. The registered format feature point coordinate table 503 is stored in the memory 50, and an index representing a document for identifying each registered format, and an index representing a feature point for identifying a feature point calculated in each registered format. , Each feature point is composed of columns of coordinates (x coordinate, y coordinate) on the registration format.

  As shown in FIG. 12, for example, the registration format whose index representing the document is ID1 has feature points f11, f12, f13, f14,..., And the coordinates of the feature points on each registration format are (x11, y11), (x12, y12), (x13, y13), (x14, y14),. The same applies to the other registration format IDs 2.

  The voting processing unit 222 searches the hash table 502 of the registration format stored in the memory 50 based on the hash value (feature amount) calculated by the feature amount calculation unit 221, and if the hash values match, the hash value Vote for an index representing a document registered in (i.e., an image for which a matching hash value has been calculated). The result of cumulative addition of votes is output to the similarity determination processing unit 223 as the number of votes.

  In addition, when the hash value of the document image and the hash value of the registered format match, the voting processing unit 222 determines which feature point of the registered format the feature point of the document image has voted, and determines the determination result. The feature point voting table 504 is stored in the memory 50.

  FIG. 13 is an explanatory diagram showing the structure of the feature point voting table 504. The feature point voting table 504 includes indexes f11, f21,..., Fn1 (hereinafter referred to as “fn1”) representing the feature points of the registration format corresponding to the indexes ID1, ID2, ID3, ID4,. .., F12, f22,..., Fn2 (hereinafter referred to as “fn2”), and similarly, the feature points of the original image are provided in each element of the matrix composed of fn3, fn4, fn5, fn6, fn7,. Indexes p1, p2,... Represented are recorded.

  As shown in FIG. 13, for example, it is determined that the hash value calculated for the feature point of the document image represented by the index p1 matches the hash value of the feature point f11 of the index ID1 representing the document in the registered format. ing. Further, it is determined that the hash value calculated for the feature point of the document image represented by the index p2 matches the hash value of the feature point f32 of the index ID3 representing the document in the registered format. Furthermore, it is determined that the hash value calculated for the feature point of the document image represented by the index p3 matches the hash value of the feature point f13 of the index ID1 representing the document in the registered format. The same applies to the indexes p4, p5, p6, p7,.

  Based on the voting result input from the voting processing unit 222, the similarity determination processing unit 223 determines which registered format the read document image is similar to, and outputs the determination result to the writing area extraction unit 224. To do. More specifically, the similarity determination processing unit 223 calculates the number of votes input from the voting processing unit 222 by the maximum number of votes of the document image (the number of feature points × the number of hash values calculated from one feature point). The similarity is calculated by dividing by (represented). The similarity determination processing unit 223 compares the calculated similarity with a predetermined threshold Th (for example, 0.8). If the similarity is equal to or greater than the threshold Th, the registration for which the similarity is calculated If it is determined that the format is similar and the similarity is smaller than the threshold Th, it is determined that there is no registered format similar to the document image, and the determination result is output to the writing area extraction unit 224. The number of votes obtained from the voting processing unit 222 is compared with a predetermined threshold. If the number of votes is equal to or greater than the threshold, it is determined that the document image is similar to a registered format registered in advance, and is further determined to be similar. It can also be determined that the registered format with the highest number of votes matches the original image.

  FIG. 14 is an explanatory diagram illustrating an example of similarity determination based on a vote result. As shown in FIG. 14, as a result of searching and voting the hash table 502 for each hash value calculated from the read document image, the index representing the document is voted for the registration formats indicated by ID1, ID2, and ID3. Suppose. The normalized similarities N1, N2, and N3 are calculated by dividing the number of votes obtained as a result of the voting by the maximum number of votes of the document image. The calculated similarity is equal to or greater than the threshold Th because the index representing the document is in the registration format indicated by ID1, and thus the document image is determined to be similar to the registration format of index ID1 representing the document.

  When the determination result that the document image is similar to the registered format is output from the similarity determination processing unit 223, the writing area extracting unit 224 determines whether or not the document image is written, and the writing is performed. In this case, the original written image is extracted, and the extracted original written image is output to the registration control unit 225 together with a determination signal indicating the presence / absence of writing. More specifically, the writing area extraction unit 224 performs alignment processing for aligning the document image and the registration format, and calculates a difference between pixel values (for example, luminance values) of each pixel of the document image and the registration format. A difference process to be calculated, a determination process for determining whether or not a document image has been written based on the calculated difference, and the like are performed.

  Next, an alignment process for aligning images by converting the coordinate system of the registered format into the coordinate system of the document image will be described. When the registration format coordinate system is converted to the original image coordinate system and image alignment is performed, the entire registration format can be coordinate-converted, or the coordinate conversion location can be set for each entry column of the registration format. However, it is also possible to convert the coordinates of only a part of the set points (or to be determined in advance). First, a case where the entire registration format is coordinate-converted and aligned with the document image will be described. In the following description, it is assumed that the document image is determined to have an index representing the document similar to the registration format of ID1.

  The writing area extraction unit 224 accesses the memory 50 and extracts the feature points of the document image whose index representing the document is voted ID1 from the feature point voting table 504. In the example of FIG. 13, the indexes representing the feature points of the document image are p1, p3, p4, and p7.

  The writing area extracting unit 224 identifies indexes f11, f13, f14, and f17 representing the feature points of the registered format corresponding to the indexes p1, p3, p4, and p7 representing the feature points of the extracted document image (hash values match). To do. The writing area extracting unit 224 refers to the feature point coordinate table 503 of the registration format and the feature point coordinate table 501 of the document image, and the matrix Jin regarding the coordinates of the feature points of the registration format represented by the formula (1) and the formula A matrix Jout is obtained for the coordinates of the feature points of the document image shown in (2), and the conversion coefficient of the conversion matrix W is assumed to be expression (3).

The writing area extraction unit 224 calculates the transformation matrix W by Expression (4). In this case, both sides of Equation (4) are multiplied by Jin transpose matrix Jin ^T to obtain Equation (5), and both sides of Equation (5) are multiplied by the inverse matrix of Jin ^T Jin to obtain Equation (6). To calculate a conversion coefficient. Using the conversion matrix W indicated by the calculated conversion coefficient, the writing area extraction unit 224 converts arbitrary coordinates (x, y) on the registration format into coordinates (x ', Y').

  FIG. 15 is an explanatory diagram showing the alignment of the document image and the registered format. As shown in FIG. 15, the registration format feature points f11, f13, f14, and f17 and the document image feature points p1, p3, p4, and p7 are associated with each other to thereby register the origin of the registration format (the upper left pixel of the image). Even if the position of the upper left pixel of the original image is shifted, the origins of both images can be made coincident, and the registration format and the original image can be aligned. Note that the number of feature points in each image is not limited to four. Depending on the format of the registration format, when there are a large number of feature points, some feature points can be selected from them.

  Next, difference processing for calculating a difference between pixel values of each pixel of the original image and the registration format will be described. The writing area extraction unit 224 sequentially scans the pixels of each of the registered original image and the registration format, for example, from the upper left to the lower right of both images, so that the luminance value of the target pixel corresponding to both images is obtained. The difference is calculated. In this case, in consideration of the reproducibility of the pixel value when reading the document, when the image is represented by 256 gradations, the difference between the luminance values of the corresponding pixels of both images is about 5-10. If the position is within the range (difference threshold Tg), it is determined that the target pixel is the same. For example, if the luminance value of a pixel of interest in a document image is 210 and the luminance value of the pixel of interest corresponding to the registration format is 215, it is determined that both pixels of interest are the same. For example, if the luminance value of a pixel of interest in the original image is 210 and the luminance value of the pixel of interest corresponding to the registration format is 235, it is determined that the pixels of interest are not the same.

  The writing area extraction unit 224 determines whether or not a ratio obtained by dividing the number of pixels determined to be the same (determination pixel number M) by the number of pixels included in the registration format is smaller than a ratio threshold Ts (for example, 0.99). If the calculated ratio is smaller than the ratio threshold value Ts, it is determined that there is writing in the document image, the written document writing image is extracted, and the extracted document writing is performed together with a determination signal indicating the presence / absence of writing. Output an image. If the calculated ratio is equal to or greater than the ratio threshold Ts, it is determined that there is no writing on the document image, and a determination signal is output. It is determined whether the ratio obtained by dividing the number of pixels determined not to be the same as the number of pixels in the registration format is greater than a ratio threshold Td (for example, 0.01), and the calculated ratio is greater than the ratio threshold Td. When the ratio is larger, it is determined that the document image is written, and when the calculated ratio is equal to or less than the ratio threshold value Td, it can be determined that the document image is not written.

  That is, the writing area extraction process calculates a difference between the acquired document image and the pixel value of the registration format (registered image), compares the calculated difference value with a predetermined threshold (difference threshold), and the pixel value is substantially omitted. Calculate the number of pixels of the same pixel (determination pixel number), calculate a ratio obtained by dividing the calculated determination pixel number by the total number of pixels in the registration format, and calculate the ratio to a predetermined threshold (ratio threshold) Are compared (large / small comparison), and the presence / absence of writing is determined according to the comparison result.

  Next, a description will be given of the case where the registration format and the document image are aligned by setting the location where coordinate conversion is performed for each registration format description column and converting the coordinates of only a part of the set location. In this case, it is determined whether or not writing is performed at a location set as a location where coordinate conversion is performed.

  FIG. 16 is an explanatory diagram showing an example of a registration format. As shown in FIG. 16A, the registration format (that is, a document in a predetermined format) is provided with a description column (writing area) for date, name, address, telephone number, and entry. Also, as shown in FIG. 16B, it is possible to set the coordinate conversion only for the name, address, and entry items (in the figure, with a pattern) in each entry column of the registration format. .

  The location where the coordinate conversion is performed may be set in advance for each registration format, and the configuration information (coordinate conversion target table in the registration format) may be stored in the memory 50, or by the user operating the operation panel 4, For example, it can be configured so that it can be set for each description column such as name, address, and entry.

  FIG. 17 is an explanatory diagram showing an example of the coordinate conversion target table 505 in the registration format. As shown in FIG. 17A, the coordinate conversion target table 505 includes an index representing a document in a registration format, a description column of the registration format, and coordinates in a description column. Further, as shown in FIG. The positions on the registration format of the description column are specified by the coordinates on the registration format at the four corners of the description column. In FIG. 17B, as an example, the coordinates (x11, y11), (x12, y11), (x11, y12), (x12, y12) of the name column in the registration format description column whose index representing the document is ID1. ).

  As shown in FIG. 17, for example, in the registration format in which the index representing the manuscript is ID1, the description columns of name, address, and entry are subject to coordinate conversion, and the name description column is on the coordinates of the registration format. It is specified by diagonal coordinate values (x11, y11), (x12, y12).

  In this case, the writing area extraction unit 224 accesses the memory 50 and extracts the feature point of the document image in which the index representing the document is voted ID1 from the feature point voting table 504. The writing area extraction unit 224 identifies an index that represents a feature point of the registered format that corresponds to the index that represents the feature point of the extracted document image (has the same hash value). The writing area extraction unit 224 refers to the feature point coordinate table 503 of the registration format, the feature point coordinate table 501 of the document image, and the coordinate conversion target table 505, and based on the feature points in the coordinate conversion target, as described above. Thus, the coordinates (x, y) in the description column on the registration format are converted into the coordinates (x ′, y ′) on the document image. Further, the writing area extraction unit 224 performs difference processing for calculating a difference between pixel values (for example, luminance values) of each pixel of the original image and the registration format in the description column, and writes the original image based on the calculated difference. Judgment processing is performed to determine whether or not.

  As a result, it is possible to perform the determination process by converting the coordinates of only the portion for determining the presence / absence of writing, and it is possible to reduce the processing effort and to speed up the writing determination process compared to the case of processing the entire image. Can be realized.

  The registration control unit 225 registers the registration format (registered image) or the registration format when the user selects the registration mode by operating the operation panel 4 in order to register (store) the document writing image written on the document image. Alternatively, a document writing image is registered.

  When registering a registration format (registered image), the registration control unit 225 sets an index representing a document in the registration format acquired by the color image input device 1, and the hash value obtained by the feature amount calculation unit 221. A registration format hash table 502 and a registration format feature point coordinate table 503 are created and stored in the memory 50 based on an index representing the feature points of the registration format, coordinates of the feature points on the registration format, and the like. As a result, information related to the registration format is registered (stored).

  In addition, when registering a document writing image, the registration control unit 225 writes the document writing extracted by the writing area extraction unit 224 into an index representing a document in a registration format determined to be similar by the similarity determination processing unit 223. The document writing image is output to the compression / decompression processing unit 226 in association with the image.

  The compression / decompression processing unit 226 applies a predetermined compression method to the original writing image input from the registration control unit 225, for example, MH (Modified Huffman), MR (Modified Read), MMR (Modified Modified Read), JPEG. (Joint Photographic Experts Group) or the like is used to perform compression processing, and the compressed image data is stored in the memory 50.

  In addition, the compression / decompression processing unit 226 expands the image data of the written image written in a predetermined location of the registration format stored in the memory 50 and outputs the expanded written image to the search processing unit 227.

  FIG. 18 is an explanatory diagram showing an example of a written image associated with a registration format. The registration format is specified by indexes ID1, ID2,. The registration format whose index representing the manuscript is ID1, for example, has columns for date, name, address, telephone number, and entry, and each of the name, address, and entry in each entry column. In the column, identifiers Q11, Q12, and Q13 for specifying the location (writing area) are respectively defined. The writing area is an example and is not limited to this, and may be provided corresponding to all the description columns.

  The write image WG11, WG12, WG13,... Written in the write area is stored (registered) in association with the registration format whose index representing the document is ID1. For example, in the write image WG11, write images “11111”, “22222”, and “33333” are written in the write areas Q11, Q12, and Q13, respectively. Note that “11111” and the like are examples and are not limited thereto. As a result, the written image WG11 such as characters, symbols, and figures written on the original document is registered in association with the registration format indicating the format of the document. Similarly, in the writing image WG12, writing images “aaaaaa”, “bbbbbb”, and “ccccc” are written in the writing areas Q11, Q12, and Q13, respectively. Note that “aaaaaa” and the like are examples and are not limited thereto. As a result, the written image WG12 such as characters, symbols, charts and the like written on the original document is registered in association with the registration format indicating the format of the document. The same applies to the written image WG13.

  In addition, write images WG21, WG22, WG23,... Written in the write area Q21 are stored (registered) in association with the registration format whose index representing the document is ID2. For example, in the write image WG21, a write image “44444” is written in the write area Q21. This is merely an example, and the present invention is not limited to this. The same applies to the written images WG22 and WG23.

  The identifiers Q11, Q12, and Q13 can be combined into one identifier for each registration format.

  When the user operates the operation panel 4 to select the search mode, the search processing unit 227 performs search processing for extracting the original written image based on the document image. More specifically, the search processing unit 227 extracts a written image similar to the original written image extracted from the original image by searching for the written image associated with the registered format similar to the original image.

  The search processing unit 227 includes pixels of the original writing image of the original image extracted by the writing area extraction unit 224 and the writing image associated with the registration format similar to the original image, for example, from the upper left to the right of both images. By sequentially scanning downward, the difference between the luminance values of the target pixels corresponding to both images is calculated. In this case, in consideration of the reproducibility of the pixel value when reading the document, when the image is represented by 256 gradations, the difference between the luminance values of the corresponding pixels of both images is about 5-10. If the position is within the range (difference threshold Tg), the target pixel is determined to be the same. For example, if the luminance value of a pixel of interest in the original writing image is 210 and the luminance value of the corresponding pixel of interest in the writing image in the registered format is 215, it is determined that both pixels of interest are the same. For example, if the luminance value of a pixel of interest in the original writing image is 210 and the luminance value of the corresponding pixel of interest in the registration format writing image is 235, it is determined that the pixels of interest are not the same.

  The search processing unit 227 divides the number of pixels determined to be the same (determination pixel number M) by the number of pixels included in the original writing image, where Rn (n is the number of writing images associated with one registered format) ) And the written image corresponding to the largest value among the calculated ratios Rn is extracted as matching the original written image. The search processing unit 227 outputs the extracted written image to the synthesis processing unit 228 together with an index representing the document in the registered format. Note that the extracted written image can be displayed on the operation panel 4 before being output to the composition processing unit 228 to prompt the user to confirm. Further, the calculated ratio Rn is compared with a predetermined ratio threshold value, and the writing images corresponding to the ratio Rn larger than the ratio threshold value are sequentially displayed or displayed as thumbnails so that the user can select them. it can.

  FIG. 19 is an explanatory diagram showing an example of a written image to be extracted. FIG. 19A shows a document image of the document PG1 read by the color image input device 1. FIG. The user reads the original PG1 with the color image input apparatus 1 and performs predetermined processing with the color image processing apparatus 2 in order to extract a written image that matches the original written image written on the original PG1, thereby performing FIG. The written image WG11 shown in (b) can be extracted.

  In addition, when searching for and extracting a writing image similar to the original writing image extracted from the original image, the search processing unit 227 extracts the original writing image by excluding a part of the original image, It is possible to search by excluding the written image in a part of the corresponding area of the registration format. For example, in the example shown in FIG. 18, a document writing image extraction process and a registration format writing image search process can be performed by excluding the writing area Q11 of the registration format whose index representing the document is ID1. As a result, even if a part of the document is missing or dirty, the original written image can be extracted by excluding that part (writing area).

  The writing area to be excluded includes, for example, displaying the specified registered format on the operation panel 4 when the registered format similar to the original image is specified by comparing the original image with the registered format, and the displayed registered format. It is possible to configure the user to specify which of the writing areas to be excluded. The user can designate a writing area that excludes a writing area corresponding to a missing part or a dirty part of the document.

  FIG. 20 is an explanatory diagram showing an example of a write image extracted when a part of the write area is excluded. FIG. 20A shows a document image of the document PG2 read by the color image input apparatus 1, and there are omissions or dirt in the name column. The user causes the color image input device 1 to read the original document PG2 in order to obtain an original document image in which missing or smudges are repaired, and the color image processing device 2 performs a predetermined process. In this case, based on the address and entry fields (writing area), extraction of the original writing image and search and extraction of the writing image of the registration format are performed, and the writing image WG11 shown in FIG. 20B is extracted. be able to.

  The composition processing unit 228 combines the registered format and the extracted written image based on the written image input from the search processing unit 227 and the index representing the document in the registered format, and outputs composite image data. As a result, the original document can be restored.

  Next, the operation of the image forming apparatus 100 will be described. FIG. 21 is a flowchart showing a procedure for registering a document writing image written on a document. The registration process is not only configured with a dedicated hardware circuit, but a personal computer having a CPU, RAM, ROM, etc. is loaded with a computer program that defines the procedure of the registration process, and the CPU executes the computer program. Can also be performed.

  The control unit 229 determines whether or not the registration mode is set (S11). If the registration mode is not set (NO in S11), the control unit 229 continues the process of step S11 and waits until the user selects the registration mode. In the registration mode (YES in S11), the control unit 229 acquires a document image obtained from a document read by the color image input device 1 (S12).

  The control unit 229 calculates a feature point based on the acquired document image (S13), and calculates a hash value (feature amount) based on the calculated feature point (S14). The control unit 229 searches the hash table 502 of the registered format based on the calculated hash value (feature value), and votes for an index representing a document with a matching hash value (S15). In this case, the control unit 229 determines which feature point of the registered format the feature point of the document image has voted, and stores the determination result.

  The control unit 229 calculates the similarity by dividing the number of votes by the maximum number of votes of the document image (S16), and determines whether the calculated similarity is greater than the threshold Th (S17). When the similarity is larger than the threshold Th (YES in S17), the control unit 229 performs a writing area extraction process (S18). Details of the writing area extraction processing will be described later. The control unit 229 registers the extracted original writing image by compressing the original writing image and storing the original writing image in association with the index representing the original in the registration format (S19).

  The control unit 229 determines whether or not all document images have been acquired (S20). If all document images have not been acquired (NO in S20), the processing from step S12 is continued. On the other hand, when the similarity is not greater than the threshold value Th (NO in S17), the control unit 229 continues the processing from step S20. When all document images have been acquired (YES in S20), control unit 229 ends the process.

  FIG. 22 is a flowchart showing the procedure of the write area extraction process. The control unit 229 converts the coordinate system (coordinates) of the registration format determined to be similar to the document image into the coordinate system (coordinates) of the document image (S101). In this case, the entire registration format may be coordinate-converted, or only the description column in which the registration format is set may be coordinate-converted.

  The control unit 229 sets the determination pixel number M indicating the number of pixels having the same pixel value (for example, luminance value) among the pixels of the document image and the registration format to “0” (S102), and pays attention to the document image and the registration format. By scanning the pixels, the difference between the pixel values of each target pixel is calculated (S103).

  The control unit 229 determines whether or not the calculated difference is smaller than the difference threshold Tg (S104). If the difference is smaller than the difference threshold Tg (YES in S104), “1” is added to the determination pixel number M ( S105). The control unit 229 determines whether or not the processing has been completed for all the pixels of the original image and the registration format (S106). If the processing has not been completed for all the pixels (NO in S106), the control unit 229 and subsequent steps. Continue processing.

  When the difference is not smaller than the difference threshold value Tg (NO in S104), the control unit 229 continues the process from step S106. When the processing has been completed for all the pixels (YES in S106), the control unit 229 determines whether or not the ratio represented by (number of determination pixels M / number of pixels in the registered format) is smaller than the ratio threshold Ts ( S107).

  When the ratio is smaller than the ratio threshold value Ts (YES in S107), the control unit 229 determines that there is writing (S108), extracts a document writing image (S109), and ends the process. On the other hand, when the ratio is not smaller than the ratio threshold Ts (NO in S107), the control unit 229 determines that there is no writing (S110), and ends the process.

  FIG. 23 is a flowchart showing a procedure of processing for restoring the original document based on the document image. This process is not limited to a dedicated hardware circuit, but is loaded into a personal computer equipped with a CPU, RAM, ROM, etc., and a computer program that defines the processing procedure is loaded and executed by the CPU. Can also be done.

  The control unit 229 determines whether or not the search mode is set (S31). If the search mode is not set (NO in S31), the control unit 229 continues the process of step S31 and waits until the search mode is selected by the user. In the search mode (YES in S31), the control unit 229 acquires a document image obtained from a document read by the color image input device 1 (S32).

  The control unit 229 calculates a feature point based on the acquired document image (S33), and calculates a hash value (feature amount) based on the calculated feature point (S34). Based on the calculated hash value (feature value), the control unit 229 searches the hash table 502 of the registered format, and votes for an index representing a document with a matching hash value (S35). In this case, the control unit 229 determines which feature point of the registered format the feature point of the document image has voted, and stores the determination result.

  The control unit 229 calculates the similarity by dividing the number of votes by the maximum number of votes of the document image (S36), and determines whether the calculated similarity is greater than the threshold Th (S37). When the similarity is larger than the threshold Th (YES in S37), the control unit 229 displays a registration format similar to the document image on the operation panel 4, accepts selection of a writing area extraction location (S38), and writes the writing area. An extraction process is performed (S39). The writing area extraction process is the process shown in FIG.

  The control unit 229 performs a search process excluding the selected writing area (S40). Details of the search process will be described later. The control unit 229 displays the extracted written image (S41), obtains user confirmation, and synthesizes the extracted written image into a registered format to perform image synthesis (S42).

  The control unit 229 determines whether or not all document images have been acquired (S43). If all document images have not been acquired (NO in S43), the processing from step S32 is continued. On the other hand, when the similarity is not greater than the threshold Th (NO in S37), the control unit 229 continues the processing from step S43. When all the original images have been acquired (YES in S43), the control unit 229 ends the process.

  FIG. 24 is a flowchart showing the procedure of search processing. The control unit 229 reads the write image associated with the registered format from the memory 50 (S131), and performs an expansion process on the read write image (S132).

  The control unit 229 sets the determination pixel number M indicating the number of pixels having the same pixel value (for example, luminance value) among the pixels of the original written image of the original image and the written image of the registered format to “0” (S133). The pixel value difference of each pixel of interest is calculated by scanning the pixel of interest of the original written image and the registered format written image (S134).

  The control unit 229 determines whether or not the calculated difference is smaller than the difference threshold Tg (S135). If the difference is smaller than the difference threshold Tg (YES in S135), “1” is added to the determination pixel number M ( S136). The control unit 229 determines whether or not the processing has been completed for all the pixels of the original writing image and the registration format writing image (S137). If the processing has not been completed for all the pixels (NO in S137), The processing after step S134 is continued.

  When the difference is not smaller than the difference threshold value Tg (NO in S135), the control unit 229 continues the process from step S137. When the processing is completed for all pixels (YES in S137), the control unit 229 calculates a ratio Rn represented by (number of determination pixels M / number of pixels of the original writing image) (S138), and the calculated ratio Rn Is recorded (S139).

  The control unit 229 determines whether or not all of the written images have been processed (S140). If all of the written images have not been processed (NO in S140), the processing from step S131 is continued. When all the written images have been processed (YES in S140), the control unit 229 extracts a written image having the largest ratio Rn (S141), and ends the processing.

  In the above-described example of the writing area extraction process, the ratio represented by (the number of determination pixels M / the number of pixels in the registration format) is used. However, the present invention is not limited to this. By calculating the number of pixels at 22, the writing area extraction processing can be realized. In this case, the document image and the registration format are binarized using a predetermined binarization threshold, the pixel number K1 of the binarized document image and the pixel number K2 of the binarized registration format are calculated, and the number of pixels Based on whether or not the difference (K1−K2) is larger than the pixel number threshold value, the presence / absence of a document writing image can be determined. It is also possible to determine the presence / absence of a document writing image by calculating an exclusive OR using the binarized data. By adopting a configuration for calculating the number of pixels, it is possible to accurately determine whether or not writing is performed, even when there is a slight writing, regardless of the size of the document entry column (writing region). In addition, by using both the write determination based on the ratio and the write determination based on the number of pixels, the write determination accuracy can be further improved.

Embodiment 2
The original paper to be read is not limited to white paper, and recycled paper or lightly colored paper may be used. Even in such a case, the writing area can be extracted with high accuracy by performing the background removal processing of the document image. In this case, the document matching processing unit 22 can perform background removal processing. Alternatively, an input tone correction unit 23 is provided after the shading correction unit 21 and a document matching processing unit 22 is provided after the input tone correction unit 23 so that the input tone correction unit 23 performs background removal processing. You can also When performing background removal on a document image, the registration format is also registered by performing background removal processing. Alternatively, the background removal is performed on the document image so that the background has the same level as the background of the registered format. This is possible by appropriately setting a density correction table for performing background removal.

  The document collation processing unit 22 extracts a signal obtained by correcting and inverting the G signal from the RGB signals input from the shading correction unit 21, and for example, divides the density of 256 levels into 16 sections and generates a density histogram of the document image. create.

  In addition, the maximum density value determined to be the background is stored in advance in the memory 50 as the first threshold value, and how many pixels or more are determined to be the background, that is, the background The minimum value of the number of pixels determined to be is stored in advance in the memory 50 as the second threshold value. Also, a density correction table for removing the background corresponding to the density classification determined to be the background is stored in the memory 50.

  The document matching processing unit 22 searches from the high density side of the created density histogram, and is in a density category equal to or lower than the first threshold value determined to be the background, and has a density of pixels equal to or higher than the second threshold value. A section is extracted as a base. The document matching processing unit 22 performs background removal processing using a density correction table corresponding to the density classification extracted as the background. Note that a luminance signal (255-Yj) can be used instead of the G signal. Here, Yj is represented by Yj = 0.30 × Rj + 0.59 × Gj + 0.11 × Bj, where Rj, Gj, and Bj are the color components of each pixel RGB.

  FIG. 25 is a flowchart showing the procedure of the write area extraction process according to the second embodiment. The control unit 229 converts the coordinate system (coordinates) of the registered format determined to be similar to the document image into the coordinate system (coordinates) of the document image (S151). In this case, as in the first embodiment, the entire registration format may be coordinate-converted, or only the description column in which the registration format is set may be coordinate-converted.

  The control unit 229 creates a density histogram of the document image based on the signal obtained by correcting and inverting the G signal extracted from the document image (S152), and extracts the background by searching from the high density side of the created density histogram. (S153). The control unit 229 removes the background of the original image using the density correction table corresponding to the density classification extracted as the background (S154).

  The control unit 229 sets the determination pixel number M indicating the number of pixels having the same pixel value (for example, luminance value) among the pixels of the document image and the registration format to “0” (S155), and pays attention to the document image and the registration format. By scanning the pixels, the difference between the pixel values of each pixel of interest is calculated (S156).

  The control unit 229 determines whether or not the calculated difference is smaller than the difference threshold Tg (S157). If the difference is smaller than the difference threshold Tg (YES in S157), “1” is added to the determination pixel number M ( S158). The control unit 229 determines whether or not the process has been completed for all the pixels of the original image and the registration format (S159). If the process has not been completed for all the pixels (NO in S159), the control unit 229 and subsequent steps. Continue processing.

  When the difference is not smaller than the difference threshold value Tg (NO in S157), the control unit 229 continues the process from step S159. When the processing has been completed for all the pixels (YES in S159), the control unit 229 determines whether or not the ratio represented by (number of determination pixels M / number of pixels in the registered format) is smaller than the ratio threshold Ts ( S160).

  If the ratio is smaller than the ratio threshold Ts (YES in S160), the control unit 229 determines that there is writing (S161), extracts the original writing image of the original image (S162), and ends the process. On the other hand, when the ratio is not smaller than the ratio threshold Ts (NO in S160), the control unit 229 determines that there is no writing (S163), and ends the process.

  As a result, even if the original paper to be read is recycled paper or lightly colored paper, the writing area can be extracted with high accuracy.

  FIG. 26 is a block diagram illustrating a configuration of the document reading apparatus 500. As shown in the figure, the document reading device 500 includes a color image input device 1, an A / D conversion unit 20, a shading correction unit 21, a document matching processing unit 22, and the like. Since the color image input device 1, the A / D conversion unit 20, the shading correction unit 21, and the document collation processing unit 22 are the same as those of the image forming apparatus 100 described above, description thereof is omitted.

  As described above, according to the present invention, it is possible to extract a written image written on an original more easily than in the past. In addition, even when a part of the document is soiled or missing, a written image written on the document can be extracted. In addition, the original document that has been written can be restored. Further, by removing the background, it is possible to accurately extract the written image written on the original. Furthermore, the original writing image written on the original can be stored in association with the registered image.

  In the above-described embodiment, the registration format and the hash table 502 are stored in the memory 50 in advance. The registration format may be stored in the storage unit of the server device, and the hash table 502 may be distributed and stored in the memory 50.

  In the above-described embodiment, as the color image input device 1, for example, a flat bed scanner, a film scanner, a digital camera, a mobile phone, or the like is used. Further, as the color image output device 3, for example, an image display device such as a CRT display or a liquid crystal display, an electrophotographic system or an ink jet system printer that outputs processing results to recording paper or the like is used. Further, the image forming apparatus 100 may include a modem as a communication unit for connecting to a server apparatus or the like via a network. Further, instead of acquiring the color image data from the color image input device 1, the color image data may be acquired from an external storage device, a server device, or the like via a network.

  In the above-described embodiment, the document collation processing unit 22 includes the memory 50 and the control unit 229. However, the present invention is not limited to this, and the memory 50 and the control unit 229 are included in the document collation processing unit 22. The structure provided outside may be sufficient.

  The present invention relates to a computer program for performing registration processing for document collation, writing area extraction processing, document restoration processing, search processing, etc. on a computer-readable recording medium on which a computer program to be executed by a computer is recorded. Program code can also be recorded. As a result, a recording medium on which the program code of the computer program for performing the above processing is recorded can be provided in a portable manner. The recording medium may be a non-illustrated memory, for example, a program medium such as a ROM because processing is performed by a microcomputer, and a program reading device as an external storage device (not illustrated) is provided, and the recording medium is stored therein. It may be a program medium that can be read by being inserted.

  In any case, the stored program code may be configured to be accessed and executed by the microprocessor, or the program code is read, and the read program code is a program (not shown) of the microcomputer. The program code may be downloaded to the storage area and executed. In this case, it is assumed that the computer program for download is stored in the main device in advance.

  Here, the program medium is a recording medium configured to be separable from the main body, such as a tape system such as a magnetic tape or a cassette tape, a magnetic disk such as a flexible disk or a hard disk, and a CD-ROM / MO / MD / DVD. Semiconductors such as optical discs, IC cards (including memory cards) / optical cards, etc., or mask ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash ROM, etc. It may be a medium that carries a computer program fixedly including a memory.

  In this case, since the system configuration is capable of connecting a communication network including the Internet, the medium may be a medium that dynamically carries the program code so as to download the program code from the communication network. When the program code is downloaded from the communication network in this way, the computer program for downloading may be stored in the main device in advance or may be installed from another recording medium. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

1 is a block diagram illustrating a configuration of an image forming apparatus including an image processing apparatus according to the present invention. It is a block diagram which shows the structure of a document collation process part. It is a block diagram which shows the structure of a feature point calculation part. It is explanatory drawing which shows the example of the filter coefficient of a filter process part. It is explanatory drawing which shows the example of the feature point of a connection area | region. It is explanatory drawing which shows the example of the extraction result of the feature point with respect to a character string. It is explanatory drawing which shows an attention feature point and the surrounding feature point. It is explanatory drawing which shows the example of calculation of the invariant by an attention feature point. It is explanatory drawing which shows the example of calculation of the invariant by an attention feature point. It is explanatory drawing which shows the structure of the feature point coordinate table of an original image. It is explanatory drawing which shows the structure of the hash table of a registration format. It is explanatory drawing which shows the structure of the feature point coordinate table of a registration format. It is explanatory drawing which shows the structure of a feature point vote table. It is explanatory drawing which shows an example of the similarity determination based on a vote result. It is explanatory drawing which shows the alignment of a manuscript image and a registration format. It is explanatory drawing which shows an example of a registration format. It is explanatory drawing which shows an example of the coordinate conversion object table of a registration format. It is explanatory drawing which shows the example of the writing image linked | related with the registration format. It is explanatory drawing which shows the example of the written image extracted. It is explanatory drawing which shows the example of the writing image extracted when a part of writing area | region is excluded. 6 is a flowchart illustrating a procedure for registering a document writing image written on a document. It is a flowchart which shows the procedure of a writing area extraction process. 6 is a flowchart illustrating a procedure of processing for restoring an original document based on a document image. It is a flowchart which shows the procedure of a search process. 10 is a flowchart illustrating a procedure of write area extraction processing according to the second embodiment. 1 is a block diagram illustrating a configuration of a document reading device.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Color image input device 2 Color image processing device 3 Color image output device 22 Document collation process part 50 Memory 220 Feature point calculation part 221 Feature-value calculation part 222 Voting process part 223 Similarity determination process part 224 Write area extraction part 225 Registration control Unit 226 compression / decompression processing unit 227 search processing unit 228 synthesis processing unit 229 control unit 2200 achromatic processing unit 2201 resolution conversion unit 2202 filter processing unit 2203 binarization processing unit 2204 centroid calculation unit

Claims (12)

In the image processing method for determining whether or not the obtained document image is similar to a registered image registered in advance, and extracting a written image written in the registered image based on the determination result,
A plurality of write images written in one or a plurality of locations determined for each of a plurality of registered images are stored in association with the registered image,
Determine whether the original image is similar to the registered image,
If the document image is similar to the registered image, determine whether the document image has been written;
If written, specify the original writing image written on the original image,
Determining whether the identified document writing image is similar to a writing image associated with a registered image similar to the document image;
An image processing method, wherein, when the original writing image is similar to the writing image, the writing image is extracted. Accept selection of the place where the written image of the registered image was written,
When determining whether or not the document image has been written, exclude the portion corresponding to the selected portion of the document image,
2. The image processing method according to claim 1, wherein when it is determined whether or not the specified document writing image is similar to the writing image of the registered image, the writing image at the selected portion is excluded. When it is determined that the original writing image is similar to the writing image, the registration image associated with the writing image is combined with the writing image,
The image processing method according to claim 1, wherein the combined registered image and written image are output. When determining whether or not a document image has been written, the presence or absence of a background is determined based on the density value of each pixel of the document image,
The image processing method according to claim 1, wherein when there is a ground, the ground is removed. In an image processing apparatus that determines whether or not an acquired document image is similar to a registered image registered in advance, and extracts a written image written in the registered image based on the determination result,
Means for storing a plurality of write images written in one or a plurality of locations determined for each of a plurality of registered images in association with the registered images;
Means for determining whether a document image is similar to a registered image;
Means for determining whether or not the original image is written when the original image is similar to the registered image;
Means for specifying a document writing image written on the document image, if written,
Means for determining whether the specified document writing image is similar to a writing image associated with a registered image similar to the document image;
An image processing apparatus comprising: means for extracting the written image when the original written image is similar to the written image. Accepting means for accepting selection of a place where a written image of a registered image is written;
Means for excluding a portion corresponding to the portion selected by the receiving means of the document image when determining whether or not the document image has been written;
6. The apparatus according to claim 5, further comprising: means for excluding a written image at a location selected by the accepting means when determining whether or not the specified original written image is similar to the written image of the registered image. Image processing apparatus. Means for synthesizing the written image with the registered image associated with the written image when it is determined that the original written image is similar to the written image;
The image processing apparatus according to claim 5, further comprising: a registered image and a written image synthesized by the unit. Means for determining the presence or absence of a background based on the density value of each pixel of the original image when determining whether or not the original image has been written;
The image processing apparatus according to any one of claims 5 to 7, further comprising: means for removing the background when there is a background. Means for accepting registration of an original writing image written on an original image;
Means for specifying a registered image similar to the original image when the means accepts registration of the original written image;
Means for specifying a document writing image written on the document image;
The image according to any one of claims 5 to 8, further comprising: a unit that stores an original writing image specified by the unit in association with a position determined for the registered image. Processing equipment.   An image forming apparatus comprising: the image processing apparatus according to claim 5; and an image forming unit that forms an output image based on an image processed by the image processing apparatus. apparatus. In a computer program for causing a computer to determine whether or not an acquired document image is similar to a registered image registered in advance, and to extract a written image written in the registered image based on the determination result,
Means for determining whether the original image is similar to the registered image;
Means for determining if the document image is similar to the registered image and whether the document image has been written;
Means for identifying the original writing image written in the original image when the computer is written;
Means for determining whether or not the specified original writing image is similar to the writing image written in one or a plurality of positions determined for a registered image similar to the original image;
A computer program for causing a computer to function as means for extracting a written image when the written image is similar to the written image. A computer-readable recording medium on which the computer program according to claim 11 is recorded.

Images