JP2022019257A - Information processing device, information processing method, and program - Google Patents

Abstract

To enable highly accurate character recognition processing on document images even when dealing with documents of various formats.SOLUTION: An information processing device for performing character string detection processing suitable for identifying a horizontal character string area is provided, the information processing device being configured to perform character string detection processing as it is for a document image with a horizontal line direction, and perform character string area detection processing for a document with a vertical line direction after rotating the image 90 degrees to make the line direction horizontal.SELECTED DRAWING: Figure 2

Description

The present invention relates to a technique for specifying a description area for a character string included in a document image.

Conventionally, there is a technique of reading and recognizing characters in a document image obtained by scanning a document. This technique is generally called OCR (Optical Character Recognition). The OCR processing is usually included in an image preprocessing for modifying a document image into an image suitable for OCR, a character string detection process for specifying a character description area (character string area) from the document image, and a detected character string area. It consists of a character recognition process that identifies each character. Regarding such OCR processing, Patent Document 1 describes a technique for detecting a character string area from vertical and horizontal projections of the entire image and performing character recognition on a document image obtained by scanning a general horizontally written document. It has been disclosed. Further, in Patent Document 2, the external frame of the character string is detected from the vertical and horizontal projections of the entire image of the document image obtained by scanning the business card, and the name is further based on the number of connected pixels in the external frame. A technique for identifying a part and performing character recognition is disclosed.

Japanese Unexamined Patent Publication No. 7-200733 Japanese Unexamined Patent Publication No. 6-96270

Conventionally, the result of the OCR processing is used for indexing a document image, and as one aspect of this indexing, there is a use case of scanning and storing a business card. Unlike general documents, business cards are usually small in paper size and contain a small number of characters. There is also a vertical business card in which the name and the like are written vertically.

In the above indexing, even for a document such as the above-mentioned vertical business card that is written vertically and has a small number of characters (character density is low), the character string area is appropriately detected from the document image and the character recognition is performed with high accuracy. Processing is required. However, the technique of Patent Document 1 assumes only general documents, that is, documents that are written horizontally and have a large number of characters (high character density), and can process documents such as vertical business cards with high accuracy. Can not. Further, the technique of Patent Document 2 is a technique specialized for business cards, and it is not assumed for a document having a complicated layout such as a pattern such as a company logo near the name that cannot be projected well.

The technique of the present disclosure has been made in view of the above problems, and even if the document to be processed contains a mixture of horizontal writing and vertical writing, the character string area is appropriately detected and the character recognition processing is performed with high accuracy. The purpose is to do.

The information processing apparatus according to the present disclosure includes a determination means for determining whether to write vertically or horizontally for a document image in which the characters described in the document are upright, and a process for rotating the document image. The detection means includes a rotation means for performing the above-mentioned rotation and a detection means for performing a character string detection process suitable for specifying the character string area for horizontal writing, and the detection means performs the rotation when the result of the determination is horizontal writing. The character string detection process is performed on the document image that has not been rotated by the means, and when the result of the determination is vertical writing, the character on the document image rotated by 90 degrees by the rotation means. It is characterized in that it performs column detection processing.

According to the technique of the present disclosure, a character string area can be appropriately detected even if horizontal writing and vertical writing are mixed in a document to be processed, and as a result, character recognition processing can be performed with high accuracy. ..

The figure which shows the hardware configuration of an information processing system. A flowchart showing the processing flow of the entire system. An example of a vertically written document image. An example of a horizontally written document image. The figure explaining how the character string area is detected from the vertical document image. A flowchart showing the details of post-processing.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the embodiments do not limit the present invention, and not all the configurations described in the embodiments are indispensable means for solving the problems of the present invention.

[Embodiment 1]
[Hardware configuration]
FIG. 1 is a diagram showing a hardware configuration of an information processing system according to the present embodiment. The information processing system includes a copying machine 100 and an information processing device 110.
The copying machine 100 includes a scanner 101 and a copying machine side communication unit 102. The scanner 101 scans the document and generates a document image. The copying machine side communication unit 102 communicates with an external device including the information processing device 110 via a network.

The information processing device 110 includes a system control unit 111, a ROM 112, a RAM 113, an HDD 114, a display unit 115, an input unit 116, and an information processing device side communication unit 117. The system control unit 111 is composed of an arithmetic unit such as a CPU, reads out a control program stored in the ROM 112, and executes various processes. The RAM 113 is used as a temporary storage area for the main memory, work area, etc. of the system control unit 111. The HDD 114 stores various data, various programs, and the like. The functions and processes of the information processing apparatus 110, which will be described later, are realized by the system control unit 111 reading a program stored in the ROM 112 or the HDD 114 and executing this program. The information processing device side communication unit 117 performs communication processing with an external device including the copying machine 100 via the network. The display unit 115 displays various information. The input unit 116 has a keyboard and a mouse, and accepts various operations by the user. The display unit 115 and the input unit 116 may be integrally provided like a touch panel. Further, the display unit 115 may be one that projects by a projector, and the input unit 116 may be one that recognizes the position of a fingertip with respect to the projected image by a camera.

In the present embodiment, the scanner 101 of the copying machine 100 scans a paper document such as a business card and generates a document image. The document image is transmitted to the information processing apparatus 110 by the communication unit 102 on the copying machine side. In the information processing device 110, the information processing device side communication unit 117 receives the document image and stores it in a storage device such as the HDD 114.

The hardware configuration of FIG. 1 is an example of a configuration for realizing the present embodiment. For example, some functions of the display unit 115 and the input unit 116 may be provided in the copying machine 100. Further, the copying machine 100 and the information processing apparatus 110 may be integrated.

<Overall processing flow>
Next, the operation flow of the software that realizes the COR processing for the document image in the information processing system according to the present embodiment will be described with reference to FIG. The series of processes shown in the flowchart of FIG. 2 is realized by the system control unit 111 reading a predetermined program from the ROM 112 or the like and executing the process. In the following description, the symbol "S" means a step.

First, in S201, the document image data is acquired from the HDD 114. Next, in S202, a process of determining the type of the original document is executed for the acquired document image. In this embodiment, it is determined whether the manuscript type is a business card or a non-business card. This determination is made based on the scan resolution and the document image size. For example, when the scan resolution is 300 DPI, if the long side of the document image obtained by scanning is in the range of 1040 pixels ± 5% and the short side is in the range of 615 pixels ± 5%, it is judged as a business card and it is out of the range. If it is judged as a non-business card. The method for determining the type of the original is not limited to the method based on the image size. For example, information for specifying the document type may be received from the input unit 116, or the feature amount may be calculated from the document image and determined by a discriminative model learned in advance.

Based on the determination result in S202, the processing flow branches in S203. If it is determined to be a non-business card, it transitions to S204, and if it is determined to be a business card, it transitions to S205.
In S204, general-purpose character recognition processing that does not depend on the document type is executed. When the character recognition process is completed, this process ends.

In S205, a process of correcting the inclination of the document image is executed. The document image generated by the scanner 101 is tilted depending on how the original is placed on the platen. Therefore, the tilt correction process is performed to obtain a document image without tilt. For the tilt correction process, a known method disclosed in Japanese Patent No. 4114959 may be applied.

Next, in S206, a process of determining the character direction in the document image is executed. Here, the character direction is defined as the character direction when the direction in which the characters in the document image stand upright is 0 degrees. In the document image subjected to the tilt correction processing in S205, the original may be rotated in units of 90 degrees. Therefore, a known method disclosed in Japanese Patent No. 3727971 or the like is applied to obtain the character direction of the document image.

Next, in S207, a process of rotating the document image based on the character direction determined in S206 is executed. As a result, a document image in which the characters are upright can be obtained.

Next, in S208, a process of extracting character pixels from the document image is executed. In the present embodiment, character pixels are extracted by binarization in which threshold processing is performed on the luminance value of each pixel. For binarization, a known method such as binarization of Otsu may be applied. Since the characters in the document are usually printed in a darker color than the background, the pixel whose brightness value is smaller than the threshold value is used as the character pixel. The method for extracting character pixels is not limited to the method by threshold processing. For example, it is realized by a method of extracting an image feature amount from an area in the vicinity of an arbitrary pixel of interest and inferring whether or not the pixel of interest is a character pixel based on a discriminative model learned in advance based on the feature amount. May be good.

Next, in S209, a process of determining the line direction is executed based on the character pixels extracted in S208. Here, the line direction is the direction in which the characters in the document image are lined up, and in the present embodiment, it is either the horizontal (horizontal) direction or the vertical (vertical) direction. As a discrimination method, for example, there is a method of generating projection histograms in the vertical direction and the horizontal direction for the entire document image and determining the direction in which the variance is small as the row direction. By this row direction determination process, for example, in the case of the business card image 301 shown in FIG. 3, the row direction is determined to be the vertical (vertical) direction, and in the case of the business card image 401 shown in FIG. 4, the row direction is determined to be the horizontal (horizontal) direction. Will be done. The obtained determination result is stored in the RAM 113.

In the next S210, the processing flow branches based on the determination result in S209. As a result of the determination, when the row direction is the horizontal direction, the transition is made to S211 and when the row direction is the vertical direction, the transition is made to S212.

In S211 when the line direction is the horizontal direction, a character string detection process suitable for specifying a horizontal character line is executed. Specifically, the presence or absence of black pixel clusters existing in the document image is scanned in the horizontal direction to identify a plurality of black pixel clusters in which the distance between the black pixel clusters is within a certain range, and the specified plurality of black pixel clusters are specified. The extrinsic rectangular area surrounding the black pixel block of is detected as one character string area. When the characters in the document image are arranged in the horizontal direction, the gap between the characters (distance between black pixel blocks = number of white pixels) is larger in the horizontal direction (horizontal direction) than in the vertical direction (vertical direction). It is usually narrower. Therefore, the character string area is detected from the document image by performing area analysis in which the processing parameters are adjusted so as to be suitable for specifying a group of characters written horizontally. As a specific method for detecting a character string, a known method disclosed in Japanese Patent Application Laid-Open No. 7-270733 or the like may be applied. As a result, for example, when the character string detection process is performed on the business card image 401 of FIG. 4 in which the row direction is horizontal, five horizontally long rectangular character string areas 402 to 406 are detected as shown in the lower part of the figure. It will be. The information of the detected character string area is stored in the RAM 113.

In S212 when the line direction is the vertical direction, the rotation process of rotating the document image by 90 degrees is executed prior to the character string detection process. For example, the business card image 301 of FIG. 3 having a vertical row direction is rotated by 90 degrees to generate a business card image 501 in which the top and bottom of the business card as shown in FIG. 5 are left and right. The business card image after the rotation processing in this way is stored in the RAM 113. As a result, the characters arranged vertically on the manuscript (business card) are arranged horizontally in the rotated image. As a result, the document image in the vertical direction can be treated as a pseudo document image in the horizontal direction, and the character string detection process common to the above-mentioned S211 can be applied.

In the following S213, as in the case of S211 described above, a character string detection process suitable for specifying a horizontally written character line is executed. For example, in the case of the business card image 501 of FIG. 5 in which the business card image 301 is rotated by 90 degrees, five character string regions 502 to 506 are detected as shown in the center of the figure. The information of the character string area detected in this way is stored in the RAM 113. Assuming that the character string detection process is performed on the business card image 301 before rotation shown in FIG. 3 as it is, for example, six character string regions 302 to 307 are detected as shown on the right side of the figure. In this example, it can be seen that the part of the surname "Jono" is separated into separate character string areas. When the processing target is a vertically long type and vertically written business card, the character string detection process is performed after rotating the image by 90 degrees as in the present embodiment, so that the character spacing is wide and discretely arranged like the name. It can be seen that the character string area of the existing character part can be detected more appropriately.

Next, in S214, a process of rotating the character string region detected in S213 by −90 degrees, that is, a process of rotating the character string region in the direction opposite to the direction rotated in S212 by the same angle is executed. As a result, the characters in the character string area return to the upright state. At this time, the partial image corresponding to the detected character string area may be rotated by −90 degrees. For example, a partial image of each of the character string regions 502, 503, 504, 505, and 506 may be cut out from the rotated business card image 501 shown in FIG. 5 and rotated by −90 degrees with respect to each partial image. .. Further, the coordinate information of the detected character string region may be rotated by −90 degrees, and the coordinate information after the rotation may be applied to the document image in the direction in which the character acquired in S207 is established. For example, the coordinate information rotated by −90 degrees corresponding to the character string areas 502', 503', 504', 505', and 506'in the business card image 501'in FIG. 5 is obtained, and the direction in which the characters stand upright is obtained. It is applied to 301 which is a document image of. As a result, it is possible to acquire the character string area in which the character stands upright.

In S215, the character recognition process is executed for each character string area acquired in S214. As a character recognition process, for example, there is a method of inferring a character code of a character included in a partial image corresponding to a character string area by using a trained model. The trained model is a discriminative model trained to output a character code by inputting a character image. Similarly, in S216, the character recognition process is executed for each character string area detected in S211. When the character recognition process of S216 is completed, this process ends.

In S217, post-processing is executed for the character recognition result obtained in S215. The details of this post-processing will be described later. When the post-processing is completed, this processing ends.

The above is the operation flow of the software that realizes the COR processing for the document image.

<Details of post-processing>
Subsequently, the post-processing of S217 will be described with reference to the flowchart of FIG.

In S601, the recognition result (character code) obtained in the character recognition process of S215 is divided into line units. If the recognition result includes a line feed code, it should be divided there. Further, using the position information for each character included in the recognition result divided into line units, the coordinate information of the circumscribed rectangle of each character string area after division is generated for each line unit. For example, in the case of the character string area 505 composed of a plurality of lines in the business card image 501 of FIG. 5, as shown by the reference numeral 507, the character string area 508 and 509 are divided into two character string areas 508 and 509, respectively. Coordinate information of the circumscribing rectangle corresponding to the column area is generated. The coordinate information of the obtained line-by-line character string area is stored in the RAM 113 together with the line-by-line character code.

Next, in S602, it is determined based on the character code whether or not the alphanumericals are dominant in the line-based character time column area obtained in S601. For example, as in the case of the business card image 301, the e-mail address and the URL of the homepage written on the vertically long type and vertically written business card are often described in a format in which the horizontally written one is rotated by 90 degrees. In general-purpose character recognition processing, it is common to build an identification model on the premise that the characters are in an upright state, but by learning the image features of the rotated characters separately, mail You can also recognize addresses. On the other hand, misjudgment of similar vertical writing characters may be mixed. Therefore, from the recognition result, it is determined whether or not the alphanumericals are dominant in the character line, that is, whether or not there is a high possibility that the character line is an e-mail address or a URL. For example, if five or more characters are recognized in one line and the majority of the characters included in the line are alphanumerical symbols, it may be determined that alphanumerical characters are dominant. The reason for setting the condition of at least 5 characters is based on the standard technical specifications of the URL domain and the e-mail address. If it is determined that the alphanumerical characters are dominant, the process proceeds to S603, and if not, this process ends.

In S603, the coordinate information of the character string area in line units determined to be dominant in alphanumericals is rotated by 90 degrees, and the vertical character string area is laid horizontally. At this time, a partial image of the character string region in line units may be generated, and the partial image may be rotated by 90 degrees. Further, the coordinate information rotated by 90 degrees may be applied to the document image (see the business card image 501 in FIG. 5) which is rotated by 90 degrees only for the coordinate information and rotated for detecting the character string that has already been generated. do not have. As a result, a character string area such as the character string area 510 in FIG. 5 is obtained by laying down what was originally vertical. As a result, it becomes a character string area in which the characters inside are upright. The character string area after rotating 90 degrees is stored in the RAM 113.

Next, in S604, the character recognition process is executed for the rotated character string area obtained in S603. The character recognition process executed here may be the same character recognition process as in S215 and S216 above, and a trained model is used in which the character types to be recognized are narrowed down to alphabets, numbers, symbols that can be used in e-mail addresses and URLs, and the like. Dedicated character recognition processing may be used. The obtained recognition result is stored in the RAM 113.

Next, in S605, the recognition result obtained in S604 is integrated with the recognition result obtained in the character recognition process in S215. This integration can be rephrased as a process of converting the coordinate information of the recognition result of S604 into a coordinate system of the original document image and incorporating it into the recognition result of S215. For example, in the character recognition process of S215 executed earlier, the coordinate system follows the business card image 501'of FIG. On the other hand, in the character recognition processing of S604 in the post-processing, the character string area 505'in the business card image 501'(the character string areas 508 and 509 after being divided in line units) is rotated 90 degrees and laid horizontally. Since 510 is the target, the coordinate systems do not match. Therefore, the coordinate information of the recognition result of the character string region 510 obtained in S604 is converted into the coordinate information so as to be the position corresponding to the coordinate system in 507, that is, 505'. In this way, the coordinate systems are matched, and the recognition result obtained by the post-processing and the recognition result obtained earlier are combined into one. The integration result is stored in the RAM 113.

The above is the content of the post-processing according to this embodiment.

According to the present embodiment, when the target document is, for example, a vertical type business card written vertically, the image is rotated so that the line direction is horizontal, and then the character string detection process for the horizontal direction is applied to detect the document. The character string area is returned to the direction in which the character stands upright, and the character recognition process is executed. This makes it possible to obtain highly accurate character recognition results. In addition, even if the horizontally written alphanumerical characters are arranged at an angle of 90 degrees in the vertical writing business card, in the post-processing, the character recognition processing is performed after rotating the characters so that they are in an upright state. Go again and integrate the recognition result with the previous recognition result. As a result, character recognition can be performed with high accuracy even for the horizontally written e-mail address and URL included in the vertically written business card.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (10)

A determination means for determining whether to write vertically or horizontally for a document image in which the characters described in the document are upright.
A rotation means that performs a process of rotating the document image, and
A detection means that performs character string detection processing suitable for specifying the horizontal character string area, and
Equipped with
The detection means
When the result of the determination is horizontal writing, the character string detection process is performed on the document image that has not been rotated by the rotation means.
When the result of the determination is vertical writing, the character string detection process is performed on the document image rotated by 90 degrees by the rotation means.
An information processing device characterized by this. The detection means horizontally scans the presence or absence of black pixel clusters existing in the document image to identify a plurality of black pixel clusters in which the distance between the black pixel clusters is within a certain range, and the identification is performed. The information processing apparatus according to claim 1, wherein an extrinsic rectangular region surrounding a plurality of black pixel blocks is detected as the character string region. Further, a character recognition means for performing character recognition processing on the character string area detected from the document image is provided.
The character recognition means is
When the result of the determination is vertical writing, the detected character string area is rotated by −90 degrees to perform the first character recognition process.
The information processing apparatus according to claim 1 or 2. The character recognition means is
When the character string area targeted for the first character recognition process is a character string area dominated by alphanumericals, the character string area is rotated 90 degrees to perform the second character recognition process.
The information processing apparatus according to claim 3. The character recognition means is
It has a means for generating a character string area for each line based on the result of the first character recognition process.
When alphanumericals are dominant in the generated line-by-line character string area, the character string area is rotated by 90 degrees to perform the second character recognition process.
The information processing apparatus according to claim 4. The information processing apparatus according to claim 4 or 5, wherein the character recognition means has a means for integrating the result of the first character recognition process and the result of the second character recognition process. The information processing apparatus according to any one of claims 4 to 6, wherein the second character recognition process is a character recognition process for recognizing a character type used for an e-mail address or a URL. The information processing apparatus according to any one of claims 1 to 7, wherein the document is a business card. A determination step that performs a process of determining whether to write vertically or horizontally for a document image in which the characters described in the document are upright.
A rotation step that performs a process of rotating the document image, and
A detection step that performs a character string detection process suitable for specifying a horizontal character string area,
Including
In the detection step,
When the result of the determination is horizontal writing, the character string detection process is performed on the document image that has not been rotated by the rotation means.
When the result of the determination is vertical writing, the character string detection process is performed on the document image rotated by 90 degrees by the rotation means.
An information processing method characterized by that. A program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 8.

Images