JP2023034823A - Image processing apparatus, and control method, and program for image processing apparatus - Google Patents

Abstract

To provide an image processing apparatus, an image processing control method, and a program for specifying and removing ruled lines so as not to reduce the accuracy of character recognition when handwritten characters are superimposed on the ruled lines.SOLUTION: In an information processing system comprising a reading apparatus and an information processing apparatus, an image processing apparatus is provided with extracting means for extracting pixels corresponding to handwritten characters 302 from an image, removing the extracted pixels, and obtaining a non-handwritten character image 305; generating means for generating ruled line determination information including at least a part of the pixels of the handwritten characters based on the pixels corresponding to the non-handwritten character image and the extracted handwritten characters; specifying means for specifying ruled line pixels from the ruled line determination information generated by the generating means; and removing means for removing the ruled line pixels specified by the specifying means from the non-handwritten character image 305.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image processing apparatus, an image processing apparatus control method, and a program.

2. Description of the Related Art Techniques for performing character recognition processing on document images read by a scanner or camera are known. As a method of character recognition processing, a technology is known in which printed characters and handwritten characters are separated in order to improve recognition accuracy, and recognition processing is performed specifically for each type of character such as printed characters and handwritten characters. .
Japanese Patent Application Laid-Open No. 2004-201001 discloses a technique of applying preprocessing for specifying a recognition target to a document image, and then separating handwritten characters and printed characters from the preprocessed image.

JP 2006-107534 A

However, the technique described in Patent Document 1 does not consider the case where handwritten characters are superimposed on ruled lines. Specifically, in the technique described in Patent Document 1, ruled lines that are noise in character recognition are removed in preprocessing for character recognition. An image is generated in which a part of the handwritten character is divided. If a part of the handwritten character is cut off, there is a possibility that the accuracy of the handwritten character separation process and handwritten character recognition will be lowered.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and provides a method for specifying and removing ruled lines without lowering the accuracy of character recognition when handwritten characters are superimposed on the ruled lines. It is intended to

An image processing apparatus according to the present invention includes extraction means for extracting pixels corresponding to handwritten characters from an image and obtaining a non-handwritten character image from which the extracted pixels are removed; generating means for generating ruled-line determination information including at least a portion of the pixels of the handwritten character based on the pixels corresponding to (1); and specifying means for specifying ruled-line pixels from the ruled-line determination information generated by the generating means and a removing means for removing the ruled line pixels specified by the specifying means from the non-handwritten character image.

According to the present invention, when a handwritten character is superimposed on a ruled line, the ruled line can be identified and removed without lowering the accuracy of character recognition.

1 is a block diagram showing a configuration example of an information processing system according to a first embodiment; FIG. 4 is a flowchart showing an example of OCR processing according to the first embodiment; FIG. 4 is a diagram showing an example of a document image according to the first embodiment; FIG. 7 is a flowchart showing an example of ruled line removal processing according to the first embodiment; 8 is a flowchart illustrating an example of processing for generating ruled line determination information according to the first embodiment; FIG. 7 is a diagram showing an example of a document image generated by ruled line removal processing according to the first embodiment; FIG. 7 is a diagram showing an example of image data generated by related information acquisition processing according to the first embodiment; FIG. 11 is a flow chart showing an example of related information acquisition processing for a ruled-line candidate area according to the second embodiment; FIG. FIG. 10 is a diagram showing an example of image data generated by related information acquisition processing according to the second embodiment;

(First embodiment)
EMBODIMENT OF THE INVENTION Hereinafter, the form for implementing this invention is demonstrated using drawing.
FIG. 1 is a block diagram showing a configuration example of an information processing system according to the first embodiment.
The information processing system 1 has a reading device 100 and an information processing device 110 . The reading device 100 has a scanner 101 and a communication section 102 . A scanner 101 reads a document and generates a scanned image. The communication unit 102 communicates with an external device such as the information processing device 110 via a network.

Information processing apparatus 110 includes system control unit 111 , ROM 112 , RAM 113 , HDD 114 , display unit 115 , input unit 116 , and communication unit 117 . The information processing device 110 according to this embodiment is an example of an image processing device.
The system control unit 111 reads control programs stored in the ROM 112 and executes various processes. A RAM 113 is used as a main memory of the system control unit 111 and a temporary storage device such as a work area. The HDD 114 stores various data, various programs, and the like. Functions and processes of the information processing apparatus 110, which will be described later, are realized by the system control unit 111 reading a program stored in the ROM 112 or the HDD 114 and executing the program.

A communication unit 117 communicates with an external device such as the reading device 100 via a network. The display unit 115 displays various information. The input unit 116 has a keyboard and a mouse, and receives various operations by the user. Note that the display unit 115 and the input unit 116 may be provided integrally like a touch panel. Further, the display unit 115 may perform projection using a projector, and the input unit 116 may recognize the position of the fingertip with respect to the projected image using a camera.
In this embodiment, the scanner 101 of the reading device 100 reads a paper document such as a form and generates a scanned image. Data of the scanned image is transmitted to the information processing apparatus 110 by the communication unit 102 . In the information processing apparatus 110 , the communication unit 117 receives the data of the scanned image, and stores the received data of the scanned image in a storage device such as the HDD 114 .

Next, an example of OCR processing in this embodiment will be described using the flowchart of FIG. The processing of this flowchart is implemented by the system control unit 111 of the information processing apparatus 110 executing according to the program stored in the ROM 112 . This processing is started when the user operates the reading device 100 to scan the paper document with the scanner 101 .
First, in S<b>200 , the system control unit 111 receives, via the communication unit 117 , scanned image data that the scanner 101 scans a paper document in accordance with an instruction from the user and transmits. The system control unit 111 then stores the received data of the scanned image in a storage unit such as the HDD 114 . FIG. 3A is a diagram showing an example of a scanned image 301 obtained by the process of S200. A scanned image 301 includes handwritten characters 302 and ruled lines 303 .

In S210, the system control unit 111 reads out the scanned image stored in the storage unit such as the HDD 114, and binarizes the scanned image. Then, the system control unit 111 stores the binary image, which is the processing result, in the RAM 113 . The binarization process is a process of quantizing image data consisting of multilevel gradation values into binary data such as 0 and 1, and a known binarization method can be used. For example, a method such as a discriminant analysis method is known as a known binarization method. FIG. 3B is a diagram showing an example of a binary image 304 obtained by binarizing the scanned image 301. As shown in FIG. Handwritten characters 302 and ruled lines 303 are also binarized.
The above binarization method is just an example, and any specific method can be used as long as it can be quantized into binary data. For example, it is possible to determine thresholds for gradation values in advance and generate binary data using the thresholds.

In S220, the system control unit 111 performs handwritten character pixel extraction processing on the binary image generated in S210. Then, system control unit 111 generates a non-handwritten character image obtained by removing the extracted handwritten character pixels from the binary image and a handwritten character image composed only of handwritten character pixels, and stores them in RAM 113 . FIG. 3C is a diagram showing an example of the result of performing extraction processing on the binary image 304. As shown in FIG. The extraction process generates a non-handwritten character image 305 for the binary image 304 and a handwritten character image 306 for the binary image 304 .
Here, since the handwritten character pixels are removed from the non-handwritten character image 305, the ruled lines are divided in the portion 307 where the handwritten characters overlap the ruled lines. FIG. 3D is a diagram showing the area corresponding to the portion 307 where the handwritten characters overlap the ruled lines before and after the process of S220. When the handwritten character pixels are removed from the handwritten partial image 308 extracted from the binary image 304, a partial image 309 of the non-handwritten image is obtained, and it can be seen that the ruled lines are finely divided.

Note that handwritten character pixel extraction is a process of extracting only pixels corresponding to handwritten characters from the scanned image, and a known handwritten character pixel extraction method can be used. For example, a method using deep learning is known as a known handwritten character pixel extraction method. Note that the handwritten character pixel extraction processing using deep learning is an example, and any specific method may be used as long as handwritten character pixels can be extracted from a scanned image. For example, it is also possible to analyze connected pixels and identify connected pixels having handwritten character characteristics as handwritten character pixels. A technique of analyzing connected pixels and determining handwritten characters is disclosed in Japanese Patent Application Laid-Open No. 10-162102.

In S230, system control unit 111 performs ruled line removal processing using the handwritten character image on the non-handwritten character image generated in S220, and stores the resulting image in RAM 113 as a printed character image. Details of the ruled line removal process will be described later with reference to FIG. FIG. 3E is a diagram showing an example of a printed character image obtained as a result of the ruled line removal processing on the non-handwritten character image 305. As shown in FIG. Pixels corresponding to ruled lines are removed by ruled line removal processing.
In S240, system control unit 111 performs handwritten OCR processing on the handwritten character image generated in S220, and stores the processing result in RAM 113 as a handwritten OCR result. The handwritten OCR process is a process of identifying character codes corresponding to characters included in a handwritten character image, and a known handwritten OCR method can be used. For example, a method of recognizing handwritten characters using pattern matching is known and disclosed in Japanese Patent Application Laid-Open No. 10-124618. Note that the handwritten character recognition process described above is just an example, and any specific technique may be used as long as the process identifies the character code of the portion corresponding to the character in the image.

In S250, the system control unit 111 performs printed character OCR processing on the printed character image generated in S230, and stores the processing result in the RAM 113 as a printed character OCR result. The printed character OCR process is a process of specifying character codes corresponding to printed characters for characters included in a printed character image, and a known printed character OCR method can be used. For example, similar to handwritten OCR, a type recognition technique using pattern matching is known. Note that the above-described printed character recognition processing is just an example, and any specific method may be used as long as it is processing for specifying the character code of a portion corresponding to a character in an image.
In S260, the system control unit 111 integrates the handwritten character recognition result generated in S240 and the printed character recognition result generated in S250, and stores them in the RAM 113 as OCR results for the scanned image.

With the above processing, it is possible to apply the optimum OCR processing to each of the handwritten characters and the printed characters, even in a form in which both handwritten characters and printed characters are mixed and ruled lines exist, thereby improving the OCR accuracy. can.
In the above process, the handwritten OCR process of S240 and the printed character OCR process of S250 may be performed in a different order. The handwritten OCR process may be performed at any timing after the handwritten character pixel extraction in S220 and before the OCR result integration process in S260. Similarly, printed character OCR may be performed at any timing after the completion of ruled line removal in S230 and before the OCR result integration processing in S260. For example, even if the order of the ruled line removal processing in S230 and the handwritten OCR processing in S240 is changed, expected results can be obtained.

Next, details of the ruled line removal process in S230 of FIG. 2 will be described using the flowchart of FIG. FIG. 4 is a flowchart showing an example of ruled line removal processing in this embodiment. The processing of this flowchart is implemented by the system control unit 111 of the information processing apparatus 110 executing according to the program stored in the ROM 112 .

In S400, the system control unit 111 refers to the RAM 113 and acquires the non-handwritten character image and the handwritten character image. Further, the system control unit 111 analyzes the obtained non-handwritten character image and the handwritten character image to generate ruled line determination information and stores it in the RAM 113 . The ruled line determination information is an image composed only of ruled line candidates.

FIG. 6A is a diagram showing an example of ruled line determination information. As shown in area 601, ruled-line candidates include, in addition to ruled lines, some handwritten character pixels that overlap the ruled lines. By integrating the pixels of the handwritten character image related to the ruled line candidate in addition to the image that is the ruled line candidate in the non-handwritten character image, the ruled line on which the handwritten characters are superimposed is expressed as an image without being divided. In addition, since some of the handwritten character pixels that are extra included in addition to the ruled lines are determined not to be ruled lines by the ruled line pixel identification processing in S410, which will be described later, some of the handwritten character pixels are not included at this stage. There is no problem in being included as ruled line candidates.
Details of the ruled line determination information generation process will be described later with reference to FIG.

In S410, the system control unit 111 refers to the RAM 113 and acquires ruled line determination information. Further, the system control unit 111 analyzes the acquired ruled-line determination information to specify ruled lines to be removed, and stores the specified ruled-line information in the RAM 113 .
Ruled line pixel identification processing is performed by thinning and thickening pixels on an image that is information for determining a ruled line, and analyzing connected pixel clusters as a result of the thinning and thickening processing. . Specifically, for example, the image, which is the information for determining ruled lines, is processed in the order of thinning and thickening pixels so that components shorter than the assumed length of ruled lines disappear. That is, the thinning process is applied by the minimum value of the length assumed for the ruled line, and some of the handwritten character pixels, which are pixels other than the ruled line, are removed. After that, by applying a thickening process with the same value as the thinning size to the result of the thinning process, the ruled lines that did not disappear can be returned to the original pixel size.

Subsequently, connected pixel blocks are extracted from the image generated by the thinning/thickening process, and the shapes and sizes of the extracted connected pixel blocks are used to determine whether the connected pixel blocks can form ruled lines. determine whether For example, if a connected pixel block is elongated, it can be determined as a ruled line.
In addition, in the ruled line pixel specifying process, straight line detection can also be used for a case where the connected pixel cluster is large in size and does not have an elongated shape, such as a table ruled line. Specifically, for example, straight line detection processing is applied to the extracted connected pixel clusters, and whether or not they form a table ruled line is determined using the detected straight line length, number, and arrangement relationship.

Note that in the present embodiment, an image obtained by applying thinning/thickening processing to a non-handwritten character image is used to analyze connected pixel blocks, but thinning/thickening processing is not applied. A connected pixel block analysis may be performed on the image. For example, if there is a dashed line, the thinning process may cause the pixel blocks that make up the dashed line to disappear. can be avoided. Specifically, for example, when looking only at a single connected pixel block, even if it is a minute point, when connected pixel blocks having fluctuation characteristics are arranged in a straight line at regular intervals in the surrounding connected pixel blocks , the corresponding connected pixel block group is determined as a ruled line candidate. By analyzing connected pixel clusters, it is possible to identify ruled line candidates even in cases where ruled lines such as dashed lines are finely divided.

FIG. 6(b) is a diagram showing an example of ruled-line information specified by performing ruled-line pixel specifying processing on the ruled-line determination information shown in FIG. 6(a). It can be seen that a region 601 that partially contained handwritten character pixels in FIG. 6A is also identified as a ruled line 602 that does not have handwritten character pixels.
In S420, the system control unit 111 refers to the RAM 113 and acquires the non-handwritten character image and the ruled line information. Further, system control unit 111 removes ruled line pixels indicated by the ruled line information from the non-handwritten character image, generates a non-handwritten character image with the ruled lines removed, and stores the generated non-handwritten character image in RAM 113 .

Next, details of the ruled line determination information generation process in S400 of FIG. 4 will be described with reference to the flowchart of FIG. FIG. 5 is a flow chart showing an example of a process of acquiring information related to a ruled-line candidate area according to the present embodiment. The processing of this flowchart is implemented by the system control unit 111 of the information processing apparatus 110 executing according to the program stored in the ROM 112 .
In S500, the system control unit 111 refers to the RAM 113 and acquires a non-handwritten character image. Further, system control unit 111 analyzes the non-handwritten character image to identify regions that are candidates for ruled lines, and stores the identified ruled-line candidate regions and ruled-line candidate images including the ruled-line candidate regions in RAM 113 .

The ruled-line candidate identification process is realized by analyzing connected pixel clusters in the non-handwritten character image. Specifically, for example, a connected pixel block is extracted from a non-handwritten character image, and the shape and size of the extracted connected pixel block are used to determine whether or not the connected pixel block can be a ruled line candidate. . Similar to the ruled-line pixel specifying process in S410 of FIG. 4, if the connected pixel cluster is elongated, it can be specified as a ruled-line candidate. Also, for large connected pixel clusters, it is possible to identify whether or not they constitute table ruled lines using straight line detection in the same manner as the ruled line pixel identification processing in S410 of FIG.
Furthermore, in the ruled-line candidate specifying process, ruled-line candidates are determined from a different perspective from the ruled-line pixel specifying process in S410 of FIG. Specifically, for example, even when minute connected pixel blocks are arranged in a straight line, the relevant connected pixel block group is determined as a ruled line candidate. As a result of this process, ruled line pixels in the non-handwritten character image that have been finely segmented due to superimposition of handwritten character pixels can also be extracted as ruled line candidates.

Subsequently, the circumscribing rectangle of the connected pixel block or connected pixel block group, which is the ruled line candidate specified by the above process, is obtained, and the area obtained by setting a predetermined margin to the circumscribing rectangle is set as the ruled line candidate area.
FIG. 7A is a diagram showing an example of a ruled-line candidate image obtained by subjecting the non-handwritten character image 305 of FIG. Areas 701 and 702 indicate areas identified as ruled line candidates. A ruled line segmented by handwritten character pixels is also specified as a group of connected pixels as a ruled line candidate, and the area 702 is also set as a ruled line candidate area.

In S510, the system control unit 111 refers to the RAM 113 and acquires the ruled line candidate area and the handwritten character image. Furthermore, the system control unit 111 acquires pixel information of the handwritten character image related to the ruled-line candidate area, and stores it in the RAM 113 as related information of the ruled-line candidate area.
The pixel information acquisition process of the handwritten character image related to the ruled line candidate area is performed by acquiring the pixels of the area corresponding to the ruled line candidate area in the handwritten character image.
For example, the area of the handwritten character image corresponding to the area 702 in FIG. 7(a) becomes the related information 703 of the ruled line candidate area in FIG. 7(b). On the other hand, the handwritten character image indicated by the area 701 in FIG. 7A does not have valid black pixels, so it is not set as the pixel information of the related handwritten character image.

In S<b>520 , the system control unit 111 refers to the RAM 113 and acquires related information of the ruled-line candidate image and the ruled-line candidate area. Furthermore, the system control unit 111 integrates the ruled-line candidate image and the related information of the ruled-line candidate area, and stores the integrated result image in the RAM 113 .
FIG. 7B is a diagram for explaining an example of the integration processing of S510. In FIG. 7B, the result of integrating the ruled-line candidate image 704 corresponding to the area 702 of FIG. 7A and the related information 703 of the ruled-line candidate area corresponding to the area 702 of FIG. An image 705 is shown.
Note that it is sufficient to set the margin of the ruled-line candidate area in the present embodiment within a range that satisfies the expected width and height of the ruled line. If a wide margin is provided, there is a high possibility that handwritten character pixels with low relevance will be treated as related information, and there is a possibility that accuracy will be lowered in the process of identifying ruled line pixels shown in S410 of FIG.

As described above, according to the present embodiment, even when a handwritten character is superimposed on a ruled line, ruled line candidates are specified from a non-handwritten character image, and the specified ruled line candidate information is used to convert the handwritten character image into the non-handwritten character image. By integrating a part, ruled lines can be removed with high accuracy.

(Second embodiment)
In the first embodiment, the handwritten character image corresponding to the ruled-line candidate area is treated as related information of the ruled-line candidate area. However, instead of simply using the handwritten character image corresponding to the ruled-line candidate area as the related information, it is possible to determine whether or not the related information is finally obtained by using the relevance between the ruled-line pixels and the handwritten character pixels included in the ruled-line candidate area. may be determined.
This case will be described with reference to FIG. FIG. 8 is a flowchart showing the ruled line determination information generation process in this embodiment, and shows the details of the ruled line determination information generation process in S400 of FIG. Note that processing other than the ruled line determination information generation processing in S400 of FIG. 4 is the same as in the first embodiment, and thus description thereof is omitted.

Since S800 is the same processing as S500 in FIG. 5, description thereof is omitted.
Since S810 is the same processing as S510 in FIG. 5, description thereof is omitted.
In S820, the system control unit 111 refers to the RAM 113 and obtains pixel information of the ruled line pixels included in the ruled line candidate area and the handwritten character image related to the ruled line candidate area. Furthermore, the system control unit 111 analyzes the acquired pixel information of the handwritten character image (hereinafter simply referred to as related information in this embodiment) and determines whether to adopt it as the final related information. When the system control unit 111 determines not to adopt the related information as the final related information, the system control unit 111 deletes the related information from the RAM 113 .

The related information adoption determination process is performed using the relationship between the ruled line pixels included in the ruled line candidate area and the related information. Specifically, for example, the system control unit 111 determines whether or not the ruled line pixels and the related information are connected as pixels. If the ruled line pixel and the related information are connected as pixels, the system control unit 111 determines that they are related. On the other hand, the system control unit 111 determines that there is no relationship when the ruled line pixel and the related information are not connected as pixels.
An example of the relevant information adoption determination process will be described below with reference to FIG. 9 . FIG. 9A is a diagram showing an example of a partial image extracted from a document image where handwritten characters are superimposed on ruled lines. FIG. 9(b) is a diagram showing an example of a handwritten character image for the partial image of FIG. 9(a). FIG. 9C is a diagram showing an example of a ruled line candidate image for the partial image of FIG. 9A, including a ruled line candidate area 901 . FIG. 9(d) is a diagram showing an example of related information of the ruled line candidate area 901 acquired in S820, which also includes a part 902 of handwritten character pixel blocks that are originally unrelated to ruled lines.

In the relevant information adoption determination process, the part 902 of the handwritten character pixel block is not connected to any ruled line pixels, so the system control unit 111 determines that it is not appropriate as related information, and deletes it from the related information stored in the RAM 113 . do. FIG. 9(e) is an example of the result of applying the relevant information adoption determination process to FIG. 9(d). As described above, part 902 of the handwritten character pixel block is deleted in FIG. 9(e).
Since S830 is the same processing as S520 in FIG. 5, description thereof is omitted.

As described above, when acquiring information related to a ruled-line candidate area, the association with ruled-line pixels included in the ruled-line candidate area is used to determine whether or not to adopt a part of a handwritten character as a ruled-line candidate. It is possible to reduce the possibility of extraction. For example, as exemplified using FIG. 8, if only some of the handwritten character pixels are focused on, they may have an elongated line shape and have characteristics similar to ruled lines. If related information having characteristics similar to ruled lines is integrated and the ruled line specifying process in S410 of FIG. By applying the processing of the present embodiment to this problem, it is possible to perform processing for specifying ruled lines after removing related information unrelated to the ruled lines.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

111 system control unit

Claims (7)

Extraction means for extracting pixels corresponding to handwritten characters from an image and obtaining a non-handwritten character image from which the extracted pixels are removed;
generating means for generating ruled line determination information including at least part of the pixels of the handwritten character based on the non-handwritten character image and the extracted pixels corresponding to the handwritten character;
specifying means for specifying ruled line pixels from the ruled line determination information generated by the generating means;
a removing means for removing the ruled line pixels identified by the identifying means from the non-handwritten character image;
An image processing device having The generating means identifies a ruled-line candidate area based on the shape and size of a pixel block that is a ruled-line candidate included in the non-handwritten character image, and converts pixels of handwritten characters included in the identified ruled-line candidate area into related information. and combining the extracted related information and the non-handwritten image to generate the ruled line determination information. 3. The image processing apparatus according to claim 2, wherein said generation means further specifies, as said ruled line candidate area, a pixel block group in which pixel blocks included in said non-handwritten character image are arranged in a straight line. The generation means determines relevance of pixels related to the extracted related information to ruled-line pixels included in the ruled-line candidate region, deletes pixels determined to have no relevance from the related information, 4. The image processing apparatus according to claim 2, wherein information for determination is generated. In the relevance determination, the generating means determines that, when pixels of handwritten characters included in the ruled-line candidate area and ruled-line pixels included in the ruled-line candidate area are not connected, the unconnected handwritten character 5. The image processing apparatus according to claim 4, wherein the ruled line determination information is generated by deleting the pixels of from the related information. an extraction step of extracting pixels corresponding to handwritten characters from an image and obtaining a non-handwritten character image from which the extracted pixels are removed;
a generation step of generating ruled line determination information including at least a portion of the pixels of the handwritten character based on the non-handwritten character image and the extracted pixels corresponding to the handwritten character;
an identifying step of identifying ruled-line pixels from the ruled-line determination information generated in the generating step;
a removing step of removing ruled line pixels identified by the identifying step from the non-handwritten character image;
A control method for an image processing apparatus having an extraction step of extracting pixels corresponding to handwritten characters from an image and obtaining a non-handwritten character image from which the extracted pixels are removed;
a generation step of generating ruled line determination information including at least a portion of the pixels of the handwritten character based on the non-handwritten character image and the extracted pixels corresponding to the handwritten character;
an identifying step of identifying ruled-line pixels from the ruled-line determination information generated in the generating step;
a removing step of removing ruled line pixels identified by the identifying step from the non-handwritten character image;
A program that causes a computer to run

Images