JP7417116B2 - Information processing system, information processing method, program - Google Patents

Description

The present invention relates to an information processing system, an information processing method, and a program.

OCR (optical character recognition) exists as a tool that assists in the task of reading information from printed forms and inputting it into a system. In OCR, before recognizing characters, there is a process called character detection that detects areas in which characters are printed within a form.

There are a wide variety of documents that can be read by OCR, such as publications and business documents, and the requirements for OCR differ depending on the purpose. Among these, when detecting characters when performing OCR on documents, it is important to ensure that even short character strings are not overlooked, that headings with wide spacing between characters can be recognized as one character string, and that unrelated character strings are combined. It is required that such things be avoided.

Hybrid Page Layout Analysis via Tab-Stop Detectionhttps://static. googleusercontent. com/media/research. google. com/ja//pubs/archive/35094. pdf

Non-Patent Document 1 describes an OCR technique for text in columns such as magazines, newspapers, and papers.

On the other hand, if documents such as invoices and receipts are treated as text in columns and OCR processing is performed, a problem arises in that adjacent character strings that are unrelated to each other are combined into one paragraph. Put it away.

In addition to Non-Patent Document 1, character detection methods that apply object detection methods using machine learning have been proposed, but object detection-based methods are There is a tendency for character strings with important meanings to be overlooked (character strings with important meanings), and the above-mentioned requirements for form OCR are not met.

Therefore, an object of the present invention is to provide a technique that can obtain more appropriate character recognition results.

The information processing system of the present invention includes a continuous pixel acquisition unit that acquires consecutive pixels from an image that is a target of character recognition, and an estimation that estimates a character area based on the pixels acquired by the continuous pixel acquisition unit. and a character recognition means that executes character recognition processing in units of character areas estimated by the estimation means.

According to the present invention, it is possible to obtain more appropriate character recognition results.

1 is a diagram showing an example of a system configuration of a table extraction system in an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the hardware configuration of a PC in an embodiment of the present invention. FIG. 2 is a diagram showing an example of a functional configuration in an embodiment of the present invention. FIG. 3 is a diagram showing an example of a processing result of an image preprocessing unit in an embodiment of the present invention. It is a figure which shows an example of the detection result of a continuous pixel detection part in embodiment of this invention. FIG. 6 is a diagram illustrating an example of a processing result of an image segment classification unit in an embodiment of the present invention. It is a flowchart which shows the flow of processing of a character area estimation part in an embodiment of the present invention. FIG. 3 is a diagram showing an example of a work image in which coordinates of image pieces classified as characters are plotted in an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a process for estimating a group of characters in an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of processing for combining isolated characters in an embodiment of the present invention. It is a figure showing an example of a character detection result in an embodiment of the present invention. It is a figure showing an example of an output result in an embodiment of the present invention. It is a figure which shows an example of the process of a cell division part in embodiment of this invention. FIG. 3 is a diagram illustrating an example of processing for classifying printed characters and handwriting of a character string image in an embodiment of the present invention.

Embodiments of the present invention will be described in detail below with reference to the drawings.

FIG. 1 is a diagram showing an example of the system configuration of a character recognition system according to an embodiment of the present invention.

A client PC 101 for performing main processing of character recognition and a scanner 102 for scanning a form and converting it into an image file are connected via a communication path 100.

The communication path 100 can take the form of a wired LAN, wireless LAN, USB, etc. depending on the physical interface that the scanner 102 has.

A file server 103 may be placed on the communication path 100. As a method of importing an image scanned by the scanner 102 to the client PC 101, there is a method in which the image is directly sent from the scanner 102 to the client PC 101, and a method in which the image file imported by the scanner 102 is temporarily stored in the file server 103, and then the client PC 101 is sent from the file server 103. There are methods for extracting image files, and any method may be used.

FIG. 2 is a block diagram showing an example of the hardware configuration of an information processing apparatus applicable to the client PC 101, scanner 102, and file server 103 of the present invention.

As shown in FIG. 2, the information processing device includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, a storage device 204, an input controller 205, An audio controller 206, a video controller 207, a memory controller 208, and a communication I/F controller 209 are connected.

The CPU 201 centrally controls each device and controller connected to the system bus 200.

The ROM 202 or external memory 213 stores a BIOS (Basic Input/Output System) and an OS (Operating System), which are control programs executed by the CPU 201, as well as computer-readable and executable programs and various necessary programs for realizing this information processing method. Holds data (including data tables).

The RAM 203 functions as the main memory, work area, etc. of the CPU 201. The CPU 201 loads necessary programs and the like from the ROM 202 or the external memory 213 into the RAM 203 when executing processing, and executes the loaded programs to realize various operations.

The input controller 205 controls input from input devices such as a keyboard 210 and a pointing device such as a mouse (not shown). If the input device is a touch panel, the user can issue various instructions by pressing (touching with a finger or the like) an icon, cursor, or button displayed on the touch panel.

Further, the touch panel may be a touch panel capable of detecting positions touched by multiple fingers, such as a multi-touch screen.

Video controller 207 controls display on an external output device such as display 212. The display shall also include the display of a notebook computer that is integrated with the main body. Note that the external output device is not limited to a display, but may be a projector, for example. Furthermore, an input device is also provided for the device capable of accepting the above-mentioned touch operation.

Note that the video controller 207 can control a video memory (VRAM) for display control, and can use a part of the RAM 203 as a video memory area, or can provide a separate dedicated video memory. It is possible.

Memory controller 208 controls access to external memory 213. External memory can be an external storage device (hard disk) that stores boot programs, various applications, font data, user files, editing files, various data, etc., a flexible disk (FD), or a PCMCIA card slot connected via an adapter. Compact flash (registered trademark) memory and the like can be used.

The communication I/F controller 209 connects and communicates with external devices via a network, and executes communication control processing on the network. For example, communication using TCP/IP, a telephone line such as ISDN, a 4G line, a 5G line of a mobile phone, etc. is possible.

Note that the CPU 201 enables display on the display 212 by, for example, executing an outline font development (rasterization) process in a display information area in the RAM 203. Further, the CPU 201 allows the user to give instructions using a mouse cursor (not shown) on the display 212.

FIG. 3 is a diagram showing an example of the functional configuration of the client PC 101.

The input receiving unit 251 receives image input via the scanner 102 or the file server 103.

The image preprocessing unit 252 performs noise removal and binarization processing on the input image received by the input receiving unit 251.

The continuous pixel detection unit 253 detects continuous pixels (adjacent pixels) from the binarized image obtained by the processing by the image preprocessing unit 252, and cuts them out as image pieces. Note that portions that are not completely adjacent to each other and are separated by a few pixels and can be determined to be caused by blur may be cut out as the same image piece. That is, the continuous pixel detection unit 253 cuts out a portion that is determined (evaluated) to be a continuous (series) of connected lines as one image piece.

The image piece classification unit 254 has a function of determining and classifying whether the image piece cut out by the continuous pixel detection unit 253 is derived from characters or something else. During classification, parameters stored in the classification model 255 are used.

The character area estimating unit 256 estimates the area of the chunk of character strings on the form from the area information of the image piece classified as being of character origin by the image piece classifying unit 254.

The cell dividing unit 257 divides the character string area using the cell coordinate information when coordinate information of a table or a cell on a form is given.

The printed and handwritten character determination unit 260 has a function of classifying (specifying) whether the characters written in the character area are printed characters (first type of characters) or handwritten characters (second type of characters). During classification, parameters stored in the classification model 261 are used.

The character recognition unit 258 performs OCR processing (character recognition processing) on each character area estimated by the character area estimation unit 256, and recognizes the characters written in the area.

The result output unit 259 outputs a set of the detected character area and the characters written therein as a file.

FIG. 4 is a diagram showing an example of processing by the image preprocessing section 252.

An input image 401 is an image captured through a scanner 102 or the like.

The image preprocessing unit 252 binarizes and inverts the input image 401 into black and white, and then performs noise removal processing to generate a preprocessed image 402.

FIG. 5 is a diagram showing an example of the detection results of the continuous pixel detection section 253.

The continuous pixel detection unit 253 detects an area where white pixels are continuous from the preprocessed image 402 and cuts it out as an image piece. Image fragments consist of fragments of letters, letters, logos, ruled lines, and fragments of seals. In order to improve the accuracy of the classifier, which will be described later, image pieces are cut out to include some area surrounding continuous pixels.

Image pieces 501 to 506 are examples of image pieces cut out by the continuous pixel detection unit 253.

FIG. 6 is a diagram illustrating an example of a processing result by the image segment classification unit 254.

Image pieces 501-506 are classified into text and non-text depending on whether or not their origin is text. In the example of FIG. 6, 501, 502, and 505 are classified as characters, and 503, 504, and 506 are classified as non-characters. Items classified as non-text include logos, seals, and ruled lines.

A classification model 255 is used as a clue for classifying characters and non-characters. The classification model 255 is a learning model that stores character and non-character features through machine learning. VGG, ResNet, and the like are known as image classification models based on machine learning.

FIG. 7 is a flowchart showing the process flow of the character area estimation unit 256.

In step S701, a work image 801 used for character area estimation is generated. The work image 801 has the same size as the preprocessed image 402, and the entire image is filled with black.

In steps S702 to S705, each image piece detected by the continuous pixel detection unit 253 and classified by the image piece classification unit 254 is processed.

In step S703, it is checked whether the image piece to be processed is classified as a character. If it is classified as a character, the process moves to step S704.

If it is not classified as a character, the processing moves on to the next image fragment.

In step S704, an area where an image piece to be processed is detected is drawn on the workpiece image 801.

FIG. 8 is an example of an area (area indicated by a rectangle) in which the image piece drawn in step S704 is detected. Bounding boxes (rectangular areas) of continuous pixels detected from the preprocessed image 402 that are classified as characters are drawn on the work image 801 as white areas (rectangles). A set of drawn rectangles becomes a rectangle group 802. Here, rectangles corresponding to elements other than characters (underlines, ruled lines, etc.) are not drawn, thereby achieving the purpose of extracting only the area where characters are written from within the form.

In step S706, each rectangular area drawn in step S704 is expanded. Specifically, by expanding the white pixel area in the work image 801, filling in the gaps between the rectangles, and merging them, the elements in the characters that have been divided into sections, columns, etc. are brought together to the level of a character string.

Specifically, for expanding a rectangular area, for example, there is a method of expanding each rectangle by a predetermined number of pixels, or a method of expanding each rectangle by a proportion (such as 20%) depending on the size of the rectangle. In either case, in order to reduce the possibility of being combined with character strings in the upper and lower rows, it is desirable to spread it mainly in the horizontal direction and only slightly in the vertical direction.

In step S707, a group of continuous pixels is extracted again from the expanded rectangular group.

In step S708, the work image 801 is filled with a bounding box that includes the group extracted in step S707.

In step S709, isolated rectangles in the workpiece image are connected.

In step S710, the work image 801 is filled with a bounding box that includes the rectangles connected in step S709.

As described above, by expanding and combining areas detected in character parts or character units, it is possible to specify areas in character string units.

FIG. 9 is a diagram illustrating an example of a process for estimating a group of characters in steps S706, S707, and S708.

A region group 901 is obtained by expanding each region shown by a rectangle, and a character string candidate group 902 is obtained by filling in the bounding box of each region within the region group 901.

In steps S709 and S710, isolated rectangles in the work image 801 are connected to other rectangles and combined into one character string.

Some headings in a form have large spaces between characters, and in many cases such headings cannot be combined in step S706, resulting in one character in the heading becoming isolated. Here, it is possible to concatenate such isolated characters and integrate them into the original string.

FIG. 10 is a diagram showing an example of the process of combining isolated characters in steps S709 and S710. Here, it is assumed that isolated character string candidates 1010, 1011, and 1012 exist in the work image 801. These characters originally form one character string "Delivery Note 01" on the input image 401 in FIG. 4.

In step S709, for each character string candidate region, the aspect ratio of the rectangle is closer to 1 than a predetermined threshold (that is, it is estimated that only one character is isolated), and the rectangle is within a certain distance in the horizontal direction. Areas that satisfy the two conditions that character string candidate areas of the same height exist are connected with a straight line and processed to form continuous pixels. Specifically, the aspect ratio of the rectangle is closer to 1 than the predetermined threshold value, for example, under any of the following conditions.
- Condition: Th1>(horizontal size/vertical size of rectangle)>Th2 (Th1>1, Th2<1).
・(Long side size of rectangle/short side size) <Th3 (>1) condition.
・(Short side size of rectangle/long side size)> Th4 (<1) condition.

In FIG. 10, the combinations of character candidate rectangles 1010 to 1011, 1011 to 1010, and 1011 to 1012 correspond to the above condition. A continuous pixel area 1013 is obtained by connecting these character candidate rectangles with straight lines. The rectangle at the beginning of the second line, which is one below the character candidate rectangles 1010 and 1011, satisfies the condition that the aspect ratio is closer to 1 than the predetermined threshold, but there are characters with the same height within a certain distance in the horizontal direction. Since the condition that column candidates exist is not met, it is not subject to concatenation.

In step S710, a bounding box of a continuous pixel area is extracted and filled in for the work image 801 in this state, as in step S708. As a result, a character string area group 1001 is obtained.

In step S711, a character string image located at a position corresponding to the bounding box extracted in step S710 is acquired from the input image.

FIG. 11 is a diagram showing an example of the output result of the character detection process.

In this example, a character string area group 1001 is applied to an input image 401. By cutting out regions corresponding to bounding boxes from the input image 401, character string images 1101-1106 are obtained.

In step S712, it is classified whether the characters related to the character string image acquired in step S711 are printed characters (characters of the first type) or handwritten characters (characters of the second type). The classification is performed using parameters stored in the printed and handwritten classification model 261. A suitable example of the printed and handwritten classification model 261 is a trained model generated by learning (machine learning) the printed and handwritten characters. That is, in step S712, it is determined whether each character string image acquired in step S711 is handwritten or printed. Alternatively, it is determined whether the text is printed or handwritten. Then, classification is performed based on the determination results.

FIG. 14 is a diagram showing how the printed and handwritten character classification unit 260 classifies a character string image into printed characters and handwritten characters.

Among character string images 1401-1406, characters 1402, 1403, and 1404 are classified as printed characters, and characters 1401, 1405, and 1406 are classified as handwritten characters.

Then, the character recognition unit 258 performs character recognition processing on each classified character string image. In the character recognition process, according to the classification result in step S712, character strings classified as printed characters are recognized using a character recognition engine suitable for printed characters, and character strings classified as handwritten characters are recognized. By using different character recognition engines, such as performing character recognition using a character recognition engine suitable for handwritten characters, it is possible to obtain more appropriate character recognition results.

In addition, character recognition processing is not performed on character strings classified as printed characters, but character recognition processing is performed on character strings classified as handwritten characters. It becomes possible to extract a handwritten character string from a handwritten form without registering the written form in advance. In this case, in step S712, character string images determined to be handwritten characters are recognized based on the determination as to whether or not they are handwritten characters, without determining whether or not they are printed characters or classifying them. The character recognition process may not be performed on character string images that are not determined to be handwritten characters.

FIG. 12 is a diagram showing an example of an output result by the result output unit 259. In this embodiment, the output result 1201 is a JSON format text file that includes the ID of the character area, the rectangle coordinates, and the read text contents.

The cell dividing unit 257 performs a process of dividing a character detection result using cell information when information on a table area and a cell area is provided from the outside regarding a table in a form.

Since the character area estimating unit 256 processes only the areas classified as characters by the image segment classifying unit 254, the information on the ruled lines is lost at this point, and the output results of the character area estimating unit 256 are multiple. Strings in cells may be concatenated. In order to correct this result, a character string area spanning multiple cells is divided using cell rectangle information.

FIG. 13 is a diagram illustrating an example of processing by the cell division section.

When character detection is performed using this method using input image 1301 as input, character string images 1311 to 1313 are obtained, but among these, character string image 1312 has adjacent right-aligned and left-aligned text cells, so Strings are concatenated across cells.

When rectangular information is given from the outside as table 1320 and cells 1321-1324, character string image 1312 is divided into character string images 1314 and 1315 using that information.

In this way, for a tabular area, it is possible to specify the character string area for each cell and perform OCR processing without identifying characters written in multiple cells as one area. Become.

As described above, in the present invention, consecutive pixels are acquired from an image to be subjected to OCR, and a character area is estimated based on the acquired pixels. Then, OCR processing is performed for each estimated character area. By detecting characters from pixel pieces in this way, it is possible to prevent characters from being overlooked, compared to a method in which the area where characters exist is limited in advance and then detected.

Furthermore, by combining the estimated character regions, it is possible to reduce the possibility that one character is determined to be two characters (for example, ``bia'' and 旁 are recognized as separate characters).

Furthermore, by combining the estimated character areas, it becomes possible to recognize character strings with wide spacing between characters, such as those often seen in the title of a form, as a single character string.

In addition, as in this example, by classifying printed characters and handwritten characters after combining and dividing character regions, even though they are one character, it is possible to distinguish between handwritten characters and handwritten characters. It is possible to reduce the possibility of classification as printed text.

In addition, by dividing the estimated character area based on table area information (table area position, cell shape/position, etc.), character strings written in multiple cells are recognized as one character string. This makes it possible to reduce the number of things that happen.

The present invention can be implemented as, for example, a system, an apparatus, a method, a program, a recording medium, or the like. Specifically, the present invention may be applied to a system consisting of a plurality of devices, or may be applied to a device consisting of a single device.

Further, the program according to the present invention is a program that allows a computer to execute the processing method shown in the flowchart shown in FIG. 7, and the storage medium of the present invention stores a program that allows a computer to execute the processing method shown in FIG. Note that the program in the present invention may be a program for each processing method of each device shown in FIG.

As described above, a recording medium recording a program that implements the functions of the embodiments described above is supplied to a system or device, and the computer (or CPU or MPU) of the system or device reads the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by reading and executing.

In this case, the program itself read from the recording medium will realize the novel function of the present invention, and the recording medium on which the program is recorded constitutes the present invention.

Examples of recording media for supplying programs include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, and silicon A disk or the like can be used.

In addition, by executing a program read by a computer, not only the functions of the above-described embodiments are realized, but also the OS (operating system) etc. running on the computer are realized based on the instructions of the program. It goes without saying that this also includes a case where part or all of the processing is performed and the functions of the embodiments described above are realized by the processing.

Furthermore, after the program read from the recording medium is written to the memory of the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board It goes without saying that this also includes a case where a CPU or the like provided in a function expansion unit or the like performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

Moreover, the present invention may be applied to a system made up of a plurality of devices, or to a device made up of one device. It goes without saying that the present invention can also be applied to cases where the present invention is achieved by supplying a program to a system or device. In this case, by reading a recording medium storing a program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

Further, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. Note that all configurations that are combinations of the above-described embodiments and their modifications are also included in the present invention.

100 LAN
101 Client PC
102 Scanner 103 File server

Claims (4)

continuous pixel acquisition means for acquiring a plurality of pixel pieces formed by consecutive pixels of the same pixel value in a binarized image that is a target of character recognition;
a first specifying means for specifying, for each of the plurality of pixel pieces acquired by the continuous pixel acquisition means, a rectangular area including the area where the pixel piece is acquired;
a second specifying means for specifying a character area indicating an area for each character by extending the rectangular area specified by the first specifying means and combining it with another rectangular area;
Among the character areas specified by the second specifying means, a positional relationship in which the aspect ratio of the character area is closer to 1 than a predetermined threshold is within a certain distance in the horizontal direction and at the same height. a third specifying means for specifying a multi-character area including a plurality of characters by combining it with other specific character areas in the area;
Character recognition means that executes, for each of the plurality of character regions specified by the third identification means, recognition processing of characters included in the plurality of character regions ;
An information processing system comprising: When the multiple character area exists across multiple cells in the table area, further comprising dividing means for dividing the multiple character area at the boundaries of the multiple cells ,
2. The information processing system according to claim 1 , wherein, when a plurality of character regions are divided by the dividing means, the character recognition means executes the recognition process for each divided region. A continuous pixel acquisition step in which the continuous pixel acquisition means of the information processing system acquires a plurality of pixel pieces formed by consecutive pixels of the same pixel value in a binarized image that is a target of character recognition. and,
a first identifying step in which the first identifying means of the information processing system identifies, for each of the plurality of pixel pieces acquired in the continuous pixel acquiring step, a rectangular area including the area where the pixel piece is acquired; ,
A second specifying means of the information processing system specifies a character area indicating an area for each character by expanding the rectangular area specified in the first specifying step and combining it with another rectangular area. a second identification step;
The third specifying means of the information processing system is configured to determine, among the character areas specified in the second specifying step, a specific character area whose aspect ratio is closer to 1 than a predetermined threshold in a horizontal direction. a third specifying step of specifying a multi-character area containing a plurality of characters by combining it with another specific character area within a distance of and having the same height positional relationship;
a character recognition step in which the character recognition means of the information processing system executes, for each multiple character region specified in the third specifying step , a recognition process for characters included in the multiple character region ;
An information processing method comprising: A program for causing a computer to function as each means according to claim 1 or 2 .

Images