JP2023026170A - Image processing device, image processing system, image processing method, and program - Google Patents

Abstract

To improve estimation accuracy of a handwritten character region.SOLUTION: An information processing device is configured to: extract a processing target region in a read image; estimate a handwritten portion and a background portion in the read image; extract a contour existing in the processing target region in the read image; and control correction of estimation results on the basis of a coordinate position of the extracted contour, a coordinate position of the estimated handwritten portion, and a coordinate position of the estimated background portion.SELECTED DRAWING: Figure 9B

Description

The present invention relates to an image processing device, an image processing system, an image processing method, and a program.

In recent years, due to changes in the working environment accompanying the spread of computers, the digitization of work materials is progressing. Documents in which handwritten characters are entered are also being digitized, and techniques for extracting handwritten characters are being studied. Patent Document 1 describes extracting handwritten characters by extracting fine lines from a manuscript in which handwriting and printed characters are mixed and determining whether or not each fine line is handwritten according to the luminance distribution.

Japanese Patent Application Laid-Open No. 2010-218106

However, characters with a small number of pixels, such as parentheses, hyphens, and periods, are less likely to exhibit characteristics in luminance distribution. Therefore, the technique described in Patent Document 1 cannot accurately extract handwritten characters.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to improve the estimation accuracy of a handwritten character area.

An image processing apparatus according to the present invention includes acquisition means for acquiring a read image of a document including handwriting, first extraction means for extracting a processing target area in the read image, and handwritten portions and background portions in the read image. estimating means for estimating , second extracting means for extracting a contour existing in the processing target area in the read image, the coordinate position of the extracted contour and the estimated coordinate position of the handwritten portion and control means for controlling to correct the estimation result by the estimation means based on the estimated coordinate position of the background portion.

According to the present invention, it is possible to improve the accuracy of estimating a handwritten character area.

It is a figure which shows the whole structural example of an image processing system. It is a figure which shows the hardware configuration example of each apparatus. It is a figure which shows the functional structural example of a learning apparatus. 4 is a flowchart showing learning processing; 4 is a flowchart showing learning data generation processing; FIG. 4 is a diagram showing an example of a foreground original image; FIG. 4 is a diagram showing an example of a background original image; FIG. 5 is a diagram for explaining data generated in learning data generation processing; 4 is a flow chart showing processing executed in a utilization phase; 9B is a flow chart showing details of the processing performed in FIG. 9A; FIG. 4 is a diagram for explaining data generated by OCR processing; FIG. FIG. 4 is a diagram for explaining data generated by OCR processing; FIG. FIG. 4 is a diagram for explaining data generated by OCR processing; FIG. FIG. 4 is a diagram for explaining data generated by OCR processing; FIG. 9 is a flowchart showing correction processing according to the second embodiment; FIG. 4 is a diagram for explaining data generated by OCR processing; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Embodiment 1]
<Image processing system>
In this embodiment, using a neural network that estimates handwritten pixels learned using synthetically generated learning data, a region of handwritten characters is extracted from a handwritten form, and character recognition is performed on the content of the entry. A method of saving in a DB (database) will be described.
FIG. 1 is a diagram showing an example of the overall configuration of an image processing system according to this embodiment. The image processing system 100 includes an image processing device 101 , a learning device 102 , an image processing server 103 , a printed OCR server 104 , a handwritten OCR server 105 and a DB server 106 . Image processing device 101 , learning device 102 , image processing server 103 , type OCR server 104 , handwritten OCR server 105 , and DB server 106 are interconnected via network 107 .
The image processing apparatus 101 is a digital multi-function peripheral having a scanning function and a printing function, such as an MFP (Multi Function Peripheral).

In the learning phase of the image processing system 100, the image processing apparatus 101 scans a blank document in which only handwriting is written to generate image data (hereinafter, this image data is referred to as "foreground original image"). The image processing apparatus 101 scans a plurality of originals to obtain a plurality of foreground original images. The foreground original image is an example of the first read image. Also, the image processing apparatus 101 prints an electronic document and outputs a print manuscript. Furthermore, this print document is scanned to generate image data (this image data is hereinafter referred to as "background original image"). The image processing apparatus 101 scans a plurality of printed documents to obtain a plurality of background original images. The background original image is an example of the second read image. The image processing device 101 transmits the foreground original image and the background original image to the learning device 102 via the network 107 .
The learning device 102 accumulates the foreground original image and the background original image received from the image processing device 101, synthesizes the accumulated images, and generates learning data used when learning a neural network for extracting handwriting. . Then, the generated learning data is used to perform learning of the neural network to generate a learning result (parameters of the neural network, etc.).

In the usage phase of the image processing system 100, the image processing apparatus 101 scans a form containing handwriting and generates a read image to be processed (hereinafter, this scanned image data is referred to as "processing"). "Target Image"). The image processing device 101 is an example of an image generating device. The image processing apparatus 101 transmits the image to be processed to the image processing server 103 via the network 107 .
The image processing server 103 performs handwriting extraction on the processing target image received from the image processing apparatus 101 . At this time, the learning device 102 transmits the learning result to the image processing server 103 via the network 107 . The image processing apparatus 101 uses the learning results received from the learning apparatus 102 to make inferences with a neural network to estimate handwritten pixels in the processing target image. Then, the image processing server 103 extracts an area to be subjected to printed OCR and an area to be subjected to handwritten OCR based on the estimation result, and transmits the information of these areas together with the image to be processed to the printed OCR server 104 and the handwritten OCR. Send to server 105 . The image processing server 103 is an example of an image processing apparatus.

The printed character OCR server 104 performs OCR (Optical Character Recognition/Reader) suitable for character recognition of printed characters included in the image to be processed. The type OCR server 104 receives from the image processing server 103 an image to be processed and an area on the image to be processed that includes the type to be OCRed (hereinafter, this area is referred to as a "type target area"). to receive information about Then, OCR is performed on the printing target area in the processing target image to obtain text data. The printed character OCR server 104 transmits the text data to the image processing server 103 .
The handwritten OCR server 105 performs OCR suitable for recognizing handwritten characters on the handwritten characters included in the image to be processed. The handwritten OCR server 105 receives from the image processing server 103 an image to be processed and an area on the image to be processed that includes handwritten characters to be OCRed (hereinafter, this area is referred to as a "handwritten target area"). ) information. Then, OCR is performed on the handwriting target area in the processing target image to obtain text data. Handwritten OCR server 105 transmits the text data to image processing server 103 . The printed OCR server 104 and the handwritten OCR server 105 are examples of OCR devices.
The image processing server 103 integrates the text data received from the printed OCR server 104 and the handwritten OCR server 105 and transmits the integrated text data to the DB server 106 .
The DB server 106 stores the text data received from the image processing server 103 in the DB as information indicating the contents of the form. Information saved in the DB can be referenced from other systems.

<Hardware configuration of each device>
Next, with reference to FIG. 2, the hardware configuration of each device constituting the image processing system 100 described above will be described. FIG. 2A shows an example hardware configuration of the image processing apparatus 101 . FIG. 2B shows an example hardware configuration of the learning device 102 . FIG. 2C shows an example hardware configuration of the image processing server 103 . The hardware configuration of the printed character OCR server 104, the handwritten OCR server 105, and the DB server 106 is the same as that of the image processing server 103, and a description thereof will be omitted.

As shown in FIG. 2A, the image processing apparatus 101 has the following. It has a CPU 201 , a ROM 202 , a RAM 204 , a printer device 205 , a scanner device 206 , a document conveying device 207 , a storage 208 , an input device 209 , a display device 210 and an external interface 211 . Each device is connected by a data bus 203 so as to be able to communicate with each other.

A CPU 201 is a controller for overall control of the image processing apparatus 101 . The CPU 201 boots an OS (operating system) by a boot program stored in the ROM 202 . A controller program stored in the storage 208 is executed on this OS. A controller program is a program for controlling the image processing apparatus 101 . The CPU 201 centrally controls each device connected by the data bus 203 . A RAM 204 operates as a temporary storage area such as a main memory or a work area of the CPU 201 .

A printer device 205 prints image data on paper (recording material, sheet) under the control of the CPU 201 . There are electrophotographic printing methods using a photoreceptor drum or a photoreceptor belt, and inkjet methods in which ink is ejected from a fine nozzle array to directly print an image on paper. Under the control of the CPU 201, the scanner device 206 scans a document such as paper using an optical reading device such as a CCD, obtains electrical signal data, converts the data, and generates image data (read image). . A document conveying device 207 such as an ADF (automatic document feeder) conveys documents placed on a document platen on the document conveying device 207 to the scanner device 206 one by one.

The storage 208 is a readable and writable nonvolatile memory such as an HDD or SSD, and various data such as the aforementioned controller program are recorded here. The input device 209 is an input device including a touch panel, hard keys, and the like. The input device 209 accepts user's operation instructions. Then, instruction information including the indicated position is transmitted to the CPU 201 . A display device 210 is a display device such as an LCD or a CRT. The display device 210 displays display data generated by the CPU 201 . The CPU 201 determines which operation has been performed based on the instruction information received from the input device 209 and the display data displayed on the display device 210 . Then, according to the determination result, the image processing apparatus 101 is controlled, and new display data is generated and displayed on the display device 210 .

The external interface 211 transmits/receives various data including image data to/from an external device under the control of the CPU 201 via a network such as a LAN, a telephone line, or proximity wireless communication such as infrared rays. The external interface 211 receives PDL data from an external device such as the learning device 102 or a PC (not shown). The CPU 201 interprets the PDL data received by the external interface 211 and generates an image. The CPU 201 prints the generated image using the printer device 205 or stores it in the storage 208 . The external interface 211 also receives image data from an external device such as the image processing server 103 . The CPU 201 prints the received image data by the printer device 205, stores it in the storage 208, or transmits it to another external device via the external interface 211. FIG.

As shown in FIG. 2B, the learning device 102 includes a CPU 231, a ROM 232, a RAM 234, a storage 235, an input device 236, a display device 237, an external interface 238, and a GPU239. Each unit can transmit and receive data to and from each other via the data bus 233 .
The CPU 231 is a controller for controlling the learning device 102 as a whole. The CPU 231 activates the OS by a boot program stored in the ROM 232, which is nonvolatile memory. A learning data generation program and a learning program stored in the storage 235 are executed on this OS. As the CPU 231 executes the learning data generation program, the function of the learning data generation unit 301 (FIG. 3) is realized. Also, the CPU 231 executes the learning program to realize a function as a learning unit 302 (FIG. 3) that learns a neural network for estimating handwritten pixels. The CPU 231 controls each part via a bus such as the data bus 233 . The RAM 234 operates as a temporary storage area such as a main memory or work area of the CPU 231 .

The storage 235 is a readable and writable non-volatile memory, and stores the above-described learning data generation program and learning program, the foreground original image and the background original image generated by the image processing apparatus 101, and the learning data generation processing described later (FIG. 5). Store the learning data generated in .
The input device 236 is an input device including a mouse and keyboard. The display device 237 is the same as the display device 210 described with reference to FIG. 2(a).
The external interface 238 is similar to the external interface 211 described using FIG.
The GPU 239 is an image processor and cooperates with the CPU 231 to generate image data and learn neural networks.

As shown in FIG. 2C, the image processing server 103 includes a CPU 261, a ROM 262, a RAM 264, a storage 265, an input device 266, a display device 267, and an external interface 268. Each unit can transmit and receive data to and from each other via the data bus 263 .
A CPU 261 is a controller for controlling the entire image processing server 103 . The CPU 261 starts the OS by a boot program stored in the ROM 262, which is nonvolatile memory. An image processing program stored in the storage 265 is executed on this OS. When the CPU 261 executes this image processing program, processing shown in a flow chart to be described later is realized. The CPU 261 controls each part via a bus such as the data bus 263 .
The RAM 264 operates as a temporary storage area such as a main memory of the CPU 261 or a work area. The storage 265 is a readable and writable nonvolatile memory, and stores the image processing program described above.
The input device 266 is similar to the input device 236 described with reference to FIG. 2(b). The display device 267 is similar to the display device 210 described with reference to FIG. 2(a).
The external interface 268 is similar to the external interface 211 described using FIG.

FIG. 3 is a block diagram showing a functional configuration example of the learning device 102. As shown in FIG. The learning device 102 has the functions of a learning data generation unit 301 and a learning unit 302 . The CPU 231 develops the learning data generation program stored in the storage 235 in the RAM 234 and executes it, thereby realizing the function of the learning data generation unit 301 . Also, the CPU 231 develops the learning program stored in the storage 235 in the RAM 234 and executes it, thereby realizing a function as a learning unit 302 that learns a neural network for estimating handwritten pixels. In addition, the CPU 231 cooperates with the GPU 239 to execute part of the calculation processing executed by the learning data generation unit 301 and the learning unit 302 .
The learning data generation unit 301 generates learning data for learning the neural network.
The learning unit 302 uses the learning data generated by the learning data generation unit 301 to learn the neural network.

Next, processing executed in the learning phase by the image processing system 100 according to the present embodiment will be described with reference to FIGS. 4 to 8. FIG.
<Learning processing>
FIG. 4 is a flow chart showing the learning process. This flowchart is implemented by the learning unit 302 of the learning device 102 . This flowchart starts when the user performs a predetermined operation via the input device 209 of the image processing apparatus 101 . In this embodiment, the mini-batch method is used for neural network learning. Hereinafter, notation of each process (step) is omitted by adding S to the beginning of each process (step).

First, in S401, the CPU 231 initializes the neural network. Specifically, the CPU 231 constructs a neural network and randomly determines and initializes the values of each parameter included in the neural network. Various structures can be used for the neural network to be constructed, and for example, it can take the form of FCN (Fully Convolutional Networks).
In S402, the CPU 231 acquires learning data. The CPU 231 executes learning data generation processing to obtain a predetermined number (mini-batch size, for example 10) of learning data. Note that learning data for a mini-batch size may be acquired from a large amount of learning data generated by prior learning data generation processing. The learning data generation process will be described later with reference to FIG.
In S403, the CPU 231 calculates the error of the neural network. Specifically, the CPU 231 inputs the input image included in each learning data to the neural network to obtain an output. The output has the same image size as the input image, and as a prediction result, pixels determined to be handwritten have a value indicating handwriting, and pixels determined not to be handwritten indicate that they are not handwritten. is an image with the values shown. Then, the CPU 231 evaluates the difference between the output and the correct label image to find the error. Cross-entropy can be used as an index for the evaluation.

In S404, the CPU 231 adjusts parameters of the neural network. Specifically, the CPU 231 changes the parameter values of the neural network by the back propagation method based on the error calculated in S403.
In S405, the CPU 231 determines whether or not to end learning. This is done as follows. The CPU 231 determines whether or not the processes of S402 to S404 have been performed a predetermined number of times (eg, 60000 times). The predetermined number of times can be determined in advance by a user's operation input or the like at the start of this flowchart. When the CPU 231 determines that the number of times of processing has reached the predetermined number of times, the process transitions to S406. When the CPU 231 determines that the number of times of processing has not reached the predetermined number of times, the process transitions to S402, and the CPU 231 continues learning of the neural network.
In S406, the CPU 231 transmits the parameters of the neural network adjusted in S404 to the image processing server 103 as a learning result. After that, the processing of this flowchart ends.
According to the learning process described above, it is possible to learn a neural network that estimates handwritten pixels using learning data.

<Learning data generation processing>
Next, the learning data generation process executed in S402 of FIG. 4 will be described. FIG. 5 is a flowchart showing learning data generation processing. This flowchart is implemented by the learning data generation unit 301 of the learning device 102 . 6 to 8 are diagrams for explaining data generated in the learning data generation process.
First, in S<b>501 , the CPU 231 selects and reads out the foreground original image stored in the storage 235 . FIG. 6 shows an example of the foreground original image. The foreground original image is an image in which only handwriting is written, and is generated by scanning a document in which only handwriting is written on a blank sheet of paper with the image processing apparatus 101 . Assume that a plurality of foreground original images are recorded in the storage 235 in advance. In this step, one is randomly selected from a plurality of foreground original images. Here, it is assumed that the foreground original image 601 is selected from among the foreground original images 601 to 603 in FIG.
In S502, the CPU 231 rotates and processes the foreground original image read out in S501. The rotation angle is randomly selected and determined from a predetermined range (for example, between -10 degrees and 10 degrees).

In S503, the CPU 231 generates image data by cutting out a portion of the original foreground image (for example, the size of length×width=512×512) (this image data is hereinafter referred to as “foreground image”). The cutting position is determined randomly.
In S504, the CPU 231 scales and processes the foreground image generated in S503. The scaling factor is randomly selected and determined from a predetermined range (eg, between 50% and 150%). Furthermore, a part of the foreground image after scaling (for example, the size of length×width=256×256) is cut out from the center to update the foreground image.
In S505, the CPU 231 processes the foreground image by changing the brightness of each pixel. CPU 231 grayscales the foreground image and uses gamma correction to change the brightness of the foreground image. The gamma value is randomly selected and determined from a predetermined range (eg, between 0.1 and 10.0). An example of the foreground image at this point is shown in FIG. 8(a).

In S<b>506 , the CPU 231 selects and reads out the background original image stored in the storage 235 . FIG. 7 shows an example of the background original image. The background original image is generated by scanning a manuscript printed from an electronic document by the image processing apparatus 101 as it is. This manuscript does not include handwriting, and includes only objects such as typed characters and ruled lines that are printed on a form. In this embodiment, an electronic document similar in characteristics (type size, presence or absence of ruled lines, etc.) to the form scanned in the usage phase is used. In addition, since handwritten characters written in a form are targeted, an area in which handwritten characters are written in the form is used as a background original image. It is assumed that a plurality of background original images are recorded in the storage 235 in advance. In this step, one is randomly selected from a plurality of background original images.
In S507, the CPU 231 rotates and processes the background original image read out in S506. The rotation angle is randomly selected and determined from a predetermined range (for example, between -10 degrees and 10 degrees).
In S508, the CPU 231 cuts out a portion of the background original image (the same size as when the foreground image was cut out in S503) to generate image data (this image data is hereinafter referred to as a "background image"). The cutting position is determined randomly.
In S509, the CPU 231 scales and processes the background image generated in S508. The scaling factor is randomly selected and determined from a predetermined range (eg, between 50% and 150%). Furthermore, a part of the background image after scaling (the same size as when the foreground image was cut out in S504) is cut out from the center to update the background image.
In S510, the CPU 231 processes the background image by changing the brightness of each pixel. The CPU 231 grayscales the background image and uses gamma correction to change the brightness of the background image. The gamma value is randomly selected and determined from a predetermined range (eg, between 0.1 and 10.0). An example of the background image at this point is shown in FIG. 8(b).

Through the steps described above, a foreground image and a background image to be synthesized when generating learning data in subsequent steps are obtained. In this embodiment, the learning device 102 rotates, scales, and changes the luminance of each of the foreground image and the background image. As a result, the diversity of learning data can be obtained, and the generalization performance of a neural network that learns using the learning data can be improved. Note that the image processing performed on each of the foreground image and the background image is not limited to rotation, scaling, and luminance change. Also, any one of rotation, scaling, and luminance change may be selectively performed. Also, the learning device 102 does not use the original foreground image and the original background image in their original size, but randomly cuts out smaller partial images and uses them. As a result, efficiency is taken into consideration when the RAM 234 is loaded in the learning process and the CPU 231 and the GPU 239 refer to it. can be generated.

In S511, the CPU 231 generates a correct label image for the foreground image. First, the CPU 231 performs binarization processing on the foreground image. Pixel values that are lower than a predetermined threshold value are set to values indicating handwriting (for example, 255, and so on), and other pixel values are set to values that indicate non-handwriting (for example, 0, and so on). ) is generated as a correct label image for the foreground image. FIG. 8(c) shows an example of a correct label image generated from the foreground image of FIG. 8(a). White pixels in FIG. 8(c) are pixels having values indicating handwriting.
In S512, the CPU 231 generates an input image of learning data. The CPU 231 compares the same coordinates of the foreground image and the background image, and synthesizes the images by creating a new image using pixel values with lower luminance. An example of an input image generated by synthesizing the foreground image of FIG. 8(a) and the background image of FIG. 8(b) is shown in FIG. 8(d).
In S513, the CPU 231 associates the input image synthesized and generated in S512 with the correct label image (correct data) generated in S511, and stores them in a predetermined area of the storage 235 as learning data. The CPU 231 repeatedly executes the series of processes shown in this flowchart until the predetermined number of learning data is generated.

According to the learning data generation process as described above, it is possible to generate learning data for learning a neural network that estimates handwritten pixels.

Next, processing executed in the usage phase by the image processing system 100 according to the present embodiment will be described with reference to FIGS. 9A to 13. FIG. In the use phase, first, the image processing apparatus 101 scans a document containing printed characters and handwritten characters to generate an image to be processed. Then, the image to be processed is transmitted to the image processing server 103 to request OCR of printed characters and handwritten characters.

<OCR request processing>
FIG. 9A(a) is a flowchart showing OCR request processing. This flowchart is implemented by the CPU 201 of the image processing apparatus 101 reading out a controller program recorded in the storage 208, developing it in the RAM 204, and executing it. This flowchart starts when the user performs a predetermined operation via the input device 209 of the image processing apparatus 101 .

In S901, the CPU 201 controls the scanner device 206 and the document conveying device 207 to scan a document and generate a processing target image (read image). The image to be processed is generated as full-color (RGB 3-channel) image data. FIG. 10(a) shows an example of a document to be scanned. As shown in FIG. 10A, the document is a form such as a registration form, and handwritten characters are entered on the right side of each item in the form.
In S<b>902 , the CPU 201 transmits the processing target image generated in S<b>901 to the image processing server 103 via the external interface 211 . After that, the processing of this flowchart ends.

<OCR processing>
Next, OCR processing by the image processing server 103 will be described. The image processing server 103 receives an image to be processed from the image processing apparatus 101 and obtains text data by performing OCR on typed characters and handwritten characters included in the image to be processed. OCR for printed characters is performed by the printed OCR server 104 . OCR for handwritten characters is executed by the handwritten OCR server 105 . FIG. 9A(b) is a flow chart showing this OCR processing. This flowchart is implemented by the CPU 261 reading out an image processing program stored in the storage 265, developing it in the RAM 264, and executing it. This flowchart starts when the user turns on the power of the image processing server 103 .

First, in S921, the CPU 261 loads a neural network for estimating handwritten pixels. First, the CPU 261 constructs the same neural network as in S401 of the flowchart of FIG. Then, in S406 of the flowchart of FIG. 4, the learning results (neural network parameters) transmitted from the learning device 102 are reflected in the constructed neural network.
In S<b>922 , the CPU 261 determines whether an image to be processed has been received from the image processing apparatus 101 via the external interface 268 . If the image to be processed has been received, the process transitions to S923. If not received, the process transitions to S937. In this flowchart, it is assumed that an image to be processed obtained by scanning the document shown in FIG. 10A is received as the image to be processed.

In S923, the CPU 261 performs processing target region extraction processing on the processing target image received in S922, and extracts the handwritten character and printed character regions included in the processing target image as the processing target region. Details of the processing target region extraction processing will be described later with reference to FIG. 9B(a). Regions 1201 to 1205 in FIG. 12A show processing target regions obtained as a result of performing processing region extraction processing on the processing target image in FIG. 10A.
In S924, the CPU 261 estimates the handwritten portion of the processing target image received in S922. First, the CPU 261 grayscales the image to be processed. Then, this grayscaled image to be processed is input to the neural network constructed in S921 to estimate whether each pixel is handwritten. As a result, pixels that have the same size as the image to be processed and that are determined to be handwritten are given a value (e.g., 1) indicating that they are handwritten, and pixels that are determined not to be handwritten are given a value indicating that they are not handwritten. The image data in which the values (for example, 0) indicating are respectively recorded are obtained. Hereinafter, this image data will be referred to as an "estimation map". Pixels determined to be handwritten are referred to as "estimated handwritten pixels", and pixels determined not to be handwritten are referred to as "estimated background pixels". FIG. 10(b) shows an estimation map for the processing target image of FIG. 10(a). In FIG. 10B, pixels with a value of 0 are represented by black pixels, and pixels with a value of 1 are represented by white pixels in order to make the results easier to see. At this point, some of the hyphens and parentheses printed in the phone number entry field are erroneously determined to be handwritten.
In S925, the CPU 261 performs a correction process for correcting the handwriting determination on the estimation result of S924 to correct an error in the handwriting determination. Details of the correction process will be described later with reference to FIG. 9B(b). FIG. 10(c) shows an image representing the result of correcting the estimated map of FIG. 10(b). It can be seen that the erroneously determined portions are corrected by the correction processing.

In S926, the CPU 261 generates an image in which only the handwriting is extracted using the estimated map corrected in S925 as a mask. Specifically, first, the CPU 261 generates an image of the same size as the image to be processed, substitutes the pixel values of the image to be processed for the coordinates of the estimated handwriting pixels, and substitutes 255 for the coordinates of the estimated background pixels. . Hereinafter, this image will be referred to as a "handwritten extracted image". FIG. 11(a) shows a handwritten extracted image generated by using the corrected estimation map of FIG. 10(c) as a mask.
In S<b>927 , the CPU 261 performs processing target region extraction processing on the handwritten extracted image, and determines a region (handwritten target region) included in the handwritten extracted image and subjected to handwriting OCR. The details of this process are the same as those of S923, and will be described later with reference to FIG. 9B(a). Areas 1221 to 1224 in FIG. 12B show areas 1221 to 1224 in FIG. 12B that are obtained as a result of performing the processing target area extraction process on the extracted handwriting image in FIG. 11A.
In S<b>928 , the CPU 261 transmits the handwriting target area and the handwritten extracted image as targets for handwriting OCR to the handwriting OCR server 105 via the external interface 268 to execute handwriting OCR. CPU 261 of handwritten OCR server 105 executes handwritten OCR processing on the received handwritten OCR target. Handwritten OCR can be realized by applying known techniques.
In S<b>929 , the CPU 261 determines whether or not a handwritten OCR result has been received from the handwritten OCR server 105 . The handwritten OCR result is text data obtained by the handwritten OCR server 105 recognizing handwritten characters included in the handwritten target area. The process of S929 is repeated until the CPU 261 determines that the handwritten OCR result has been received from the handwritten OCR server 105 via the external interface 268, and when it is determined that the handwritten OCR result has been received, the process proceeds to S930.

Subsequently, in S930, the CPU 261 generates an image in which only the background is extracted using the estimated map corrected in S925 as a mask. Specifically, first, the CPU 261 generates an image of the same size as the image to be processed, substitutes the pixel values of the image to be processed for the coordinates of the estimated background pixels, and substitutes 255 for the coordinates of the estimated handwriting pixels. . Henceforth, this image is called a "background extraction image." FIG. 11(b) shows a background extraction image generated by using the corrected estimation map of FIG. 10(c) as a mask.
In S931, the CPU 261 performs processing target region extraction processing on the background extraction image, and determines a region (print target region) included in the background extraction image to be subjected to type OCR. The details of this process are the same as those of S923, and will be described later with reference to FIG. 9B(a). Regions 1241 to 1245 in FIG. 12(c) show the regions 1241 to 1245 in FIG. 12(c), which are obtained as a result of performing the processing target region extraction processing on the extracted background image in FIG. 11(b).
In S932, the CPU 261 transmits the text target region and the background extraction image obtained in S931 to the text OCR server 104 via the external interface 268 as text OCR targets, and causes the text OCR to be executed. The CPU 261 of the printed character OCR server 104 executes printed character OCR processing on the received printed character OCR target. The type OCR can be realized by applying a known technique.
In S<b>933 , the CPU 261 determines whether or not the print OCR result has been received from the print OCR server 104 . The printed character OCR result is text data obtained by the printed character OCR server 104 recognizing the printed characters included in the printed character target area. The process of S933 is repeated until the CPU 261 determines that the printed character OCR result has been received from the printed character OCR server 104 via the external interface 268. If it is determined that the printed character OCR result has been received, the process proceeds to S934.

Subsequently, in S934, the CPU 261 aggregates the OCR results for each processing target area. First, the CPU 261 directly applies the OCR result of the area to be processed obtained in S923 that includes either the handwritten area obtained in S927 or the printed character area obtained in S931 to the OCR of the area to be processed. result. In addition, in the case where the processing target region includes both the handwriting target region and the printed character target region, the handwritten OCR result and the printed character OCR result are arranged according to the positional relationship in the processing target region. OCR result.
The processing of this step will be described with reference to FIG. For example, since the processing target area 1203 includes only the handwriting target area 1221, the OCR result "Taro Tanaka" is used as the OCR result. On the other hand, since the processing target region 1205 includes handwriting target regions 1222, 1223, 1224 and printed character target regions 1244, 1245, each OCR result is arranged in the order of the original positional relationship, and "02-(32)-1268" is the OCR result. and

In S935, the CPU 261 integrates the OCR results of each processing target area. Here, for each region to be processed, the positional relationship and semantic validity are evaluated to estimate item-value pairs. For example, if the closest processing target area to the processing target area 1202 is the area 1203, and the OCR result "name" of the area 1202 is the item name, the OCR result "Taro Tanaka" of the area 1203 includes the name, so the value is is highly relevant. Therefore, the OCR result of area 1202 and the OCR result of area 1203 are presumed to be an item-value pair relating to the name. In a similar manner, assume that the OCR results for regions 1204 and 1205 are also item-value pairs, respectively.
In S<b>936 , the CPU 261 transmits the item-value pairs obtained in S<b>935 to the DB server 106 via the external interface 268 for storage.
In S937, the CPU 261 determines whether or not to end the series of processes. Unless the CPU 261 detects a predetermined operation such as powering off the image processing server 103, the process proceeds to S922. When the CPU 261 detects the predetermined operation, the processing of this flowchart ends.
According to the OCR processing as described above, an area containing only handwritten characters and an area containing only printed characters are extracted from an image to be processed in which handwritten characters and printed characters are mixed, and the results of character recognition performed on each area are stored in a DB server. 106.

<Processing target region extraction processing>
Next, the processing target area extraction processing executed in S923, S927, and S931 of FIG. 9A(b) will be described. FIG. 9B(a) is a flowchart showing the processing target area extraction process. This flowchart is implemented by the CPU 261 reading out an image processing program stored in the storage 265, developing it in the RAM 264, and executing it. This flowchart is executed in S923, S927, and S931 of FIG. 9A(b) with the processing target image, the handwritten extracted image, and the background extracted image as inputs, respectively.
First, in S951, the CPU 261 applies contraction processing to the input image. As a result, the characters are thickened, and small parts such as radicals and dots that make up the characters are connected to the surrounding characters, which can be suppressed from being treated as noise in the subsequent processing (S953).
In S952, the CPU 261 acquires the circumscribed rectangle of the area where the black pixels are connected. The CPU 261 searches for areas where black pixels are connected to the image that has been contracted in S951, and individually generates circumscribing rectangles for all the searched areas.
In S953, the CPU 261 excludes rectangles that are unlikely to be characters from the circumscribing rectangles generated in S952. For example, a predetermined range is provided for the length and area of the sides of a rectangle, and characters outside the predetermined range are presumed to be non-characters and removed. This allows the elimination of large rectangles surrounding figures and tiny rectangles surrounding small noises.
In S954, the CPU 261 connects adjacent circumscribing rectangles. For each rectangle remaining as a result of S953, if there is another rectangle within a certain distance to the left and right of the rectangle, the CPU 261 replaces all of those rectangles with a new rectangle that combines them. This makes it possible to form a rectangle that encloses a group of words or whole sentences instead of individual characters. Each rectangle obtained as a result is set as a processing target area. After that, the process returns to the flow of FIG. 9A(b).

According to the process target area extraction process described above, it is possible to extract areas such as typed characters representing entry items and handwritten characters entered in entry fields as process target areas. In S923, the CPU 261 extracts areas of handwritten characters and printed characters included in the image to be processed. In S927, the CPU 261 extracts a processing target area from the handwritten extracted image and determines the extracted area as a handwritten target area. In S931, the CPU 261 extracts a processing target area from the background extraction image and determines the extracted area as a printing target area.

<Correction processing for handwriting determination>
Next, a correction process for correcting an error in handwriting determination performed in S925 of FIG. 9A(b) will be described. FIG. 9B(b) is a flowchart showing correction processing according to the present embodiment. This flowchart is implemented by the CPU 261 reading out an image processing program stored in the storage 265, developing it in the RAM 264, and executing it. This flowchart is executed in S925 of FIG. 9A(b) with the estimated map obtained in S924 as input.

First, in S971, the CPU 261 determines whether the subsequent processes from S972 to S977 have been completed for all the processing target areas obtained in S923 of FIG. 9A(b). When the CPU 261 determines that processing has been completed for all processing target areas, the process returns to the flow of FIG. 9A(b). On the other hand, when the CPU 261 determines that there is an unprocessed processing target area, one processing target area is selected, and the process transitions to S972. In this manner, when a plurality of processing target regions are extracted from the processing target image, the CPU 261 sequentially determines whether or not to perform correction for the plurality of processing target regions. In the following description, it is assumed that the processing target area 1205 shown in FIG. 12A is selected.
In S972, the CPU 261 extracts contours present in the processing target area selected in S971. Contour extraction is performed by using a general contour tracking algorithm or the like after binarizing the image within the processing target area. FIG. 13(a) shows contours 1301 to 1312 extracted from a region 1205 to be processed. As shown in FIG. 13(a), an area surrounding the character stroke with an outline is extracted.
In S973, the CPU 261 determines whether the subsequent processes from S974 to S977 have been completed for all contours extracted in S972. When the CPU 261 determines that processing has been completed for all contours, the processing transitions to S971. On the other hand, when the CPU 261 determines that there is an unprocessed contour, one contour is selected and the process transitions to S974. In this manner, when a plurality of contours are extracted from the processing target area, the CPU 261 sequentially determines whether to correct the plurality of contours. In the following description, it is assumed that the contour 1307 in FIG. 13(a) is selected. Note that in this embodiment, when one closed region is formed by a plurality of contours, such as the contour 1301 and the contour 1312, the plurality of contours are treated as one contour.

In S974, the CPU 261 calculates a circumscribed rectangle of the contour selected in S973. A circumscribed rectangle calculated for the contour 1307 is indicated by a dotted line frame 1321 in FIG. 13(a).
In S975, the CPU 261 determines whether or not the circumscribed rectangle calculated in S974 satisfies a predetermined condition. In this embodiment, the predetermined condition is a condition indicating a circumscribing rectangle of an outline of a printed character that is likely to be confused with handwriting. Typefaces that are likely to be confused with handwriting are, for example, typeface brackets and hyphens. The shape of the circumscribing rectangles of type brackets and hyphens is tabulated in advance in various fonts and sizes, and the range of possible area and aspect ratio is statistically defined. The range of the circumscribing rectangle of the parentheses in type is, for example, 6 to 15 pixels high and 26 to 34 pixels wide. The circumscribing rectangle of a hyphen in a type is, for example, 2 to 6 pixels long and 20 to 40 pixels wide. Typographic brackets and hyphen bounding rectangles tend to have an elongated shape. When the CPU 261 determines that the circumscribing rectangle satisfies the predetermined condition, the process proceeds to S976. Transition.

In S976, the CPU 261 determines whether or not the outline selected in S973 includes both the handwritten portion and the background portion. This determination is made by comparing the coordinate position of the outline selected in S973, the coordinate position of the estimated handwriting pixel, and the coordinate position of the estimated background pixel. Specifically, it is determined whether or not both estimated handwriting pixels and estimated background pixels exist within the area surrounded by the extracted outline. FIG. 13(b) shows an image of the processing target area 1205 extracted from the estimation map shown in FIG. 10(b). In FIG. 13(b), it can be seen that both estimated handwriting pixels and estimated background pixels exist at positions corresponding to the contour 1307 in FIG. 13(a). If the CPU 261 determines that both the estimated handwriting pixels and the estimated background pixels are included in the contour, the process proceeds to S977. Contours are excluded from correction, and the process proceeds to S973 again.
In S977, the CPU 261 corrects all pixels in the contour selected in S973 to background estimated pixels. FIG. 13(c) shows the result of executing the process of this step on the estimation map shown in FIG. 13(b). As shown in FIG. 13C, the estimated handwriting pixels that existed within the contour 1307 are corrected to the estimated background pixels. After that, the process transitions to S973.

According to the above-described correction processing, the image processing server 103 estimates that there is a contour that satisfies a predetermined condition in the region to be processed, and that the contour contains pixels estimated to be handwritten and the background. If both of the estimated pixels are included, the estimation result is corrected. Specifically, the estimation result is corrected so that the entirety of the contour that satisfies conditions such as brackets and hyphens in the type indicates the background portion.

According to the first embodiment as described above, after estimating a handwritten portion, if there are both a portion estimated to be handwritten and a portion estimated to be background in strokes in the processing target area, an estimation error is can be corrected. As a result, it is possible to correct the estimated position of the handwritten part for the strokes in the processing target area, and in the handwritten image to be processed, part of the character is lost or an extra part is added. can be suppressed. That is, it is possible to improve the estimation accuracy of the handwritten character area. Furthermore, it is possible to improve the recognition accuracy of handwritten characters. Further, in the first embodiment, when the strokes in the processing target area show predetermined features such as strokes of printed characters that are likely to be confused with handwriting, the estimation results at the positions of the strokes can be corrected to the background portion. As a result, it is possible to correct errors in estimation of type strokes, such as parentheses and hyphens, which are likely to be confused with handwriting, and to further improve the estimation accuracy.

As a first modification of the present embodiment, an estimation result , the content of correction may be different from the above.

As a second modification of the present embodiment, even if a contour extracted from a region to be processed satisfies a predetermined condition and includes a mixture of estimated handwriting pixels and estimated background pixels, the contour is There may be a condition to exclude the correction target. This condition is, for example, that there are no presumed handwritten pixels around the outside of the contour.

[Embodiment 2]
This embodiment differs from the first embodiment in that, in the handwriting determination correction process, the correction content is switched according to the number of estimated handwriting pixels and the number of estimated background pixels in the contour. Note that the configuration of the image processing system 100 according to the present embodiment is the same as that of the first embodiment except for characteristic portions. Therefore, similar configurations are denoted by similar reference numerals, and detailed descriptions thereof are omitted. The following description focuses on differences from the first embodiment.

<Correction processing for handwriting determination>
The correction process for correcting an error in handwriting determination performed in S925 of FIG. 9A(b) will be described below. FIG. 14 is a flowchart showing correction processing according to this embodiment. This flowchart is implemented by the CPU 261 reading out an image processing program stored in the storage 265, developing it in the RAM 264, and executing it. This flowchart is executed in S925 of FIG. 9A(b) with an image obtained as a result of estimating handwritten pixels in S924 as an input.

S971 to S975 in FIG. 14 are the same as in the first embodiment. In the description of the present embodiment, it is assumed that an estimation map as shown in FIG. 15A is obtained as a result of estimating handwritten pixels for the image to be processed in FIG. 10A. At this point, some of the hyphens and parentheses printed in the phone number input field to assist handwriting are misjudged as handwritten, and the handwritten "1" is entered in the phone number input field. are misidentified as background.

Next, in S1401, the CPU 261 counts the number of presumed handwriting pixels and the number of presumed background pixels among the pixels at the corresponding positions within the contour selected in S973.
In S1402, the CPU 261 compares the number of estimated handwritten pixels counted in S1401 with the number of estimated background pixels, and determines whether or not the number of estimated handwritten pixels is greater than the number of estimated background pixels. If the CPU 261 determines that the number of estimated handwritten pixels is greater than the number of estimated background pixels, the process proceeds to S1404, and if the CPU 261 determines that the number of estimated handwritten pixels is less than the number of estimated background pixels, the process proceeds to S1403.
FIG. 15(b) shows an image of the processing target area 1205 extracted from the estimation map shown in FIG. 15(a). In the estimation map shown in FIG. 15(b), the number of estimated handwriting pixels is less than the number of estimated background pixels in the pixels within the area of the contour 1307 in FIG. 13(a). Therefore, the number of estimated background pixels is larger, and the process transitions to S1403. On the other hand, in the pixels within the area of the contour 1309 in FIG. 13A, the number of estimated handwriting pixels exceeds the number of estimated background pixels. Therefore, the number of handwritten estimated pixels is greater, and the process transitions to S1404. In this step, the number of estimated handwritten pixels and the number of estimated background pixels are compared, but the determination may be made by comparing with a predetermined number of pixels. Result of executing S1403 described below on the contour 1307 of FIG. 13A and executing S1404 described below on the contour 1309 of FIG. 13A in the estimation map of FIG. 15B is shown in FIG. 15(c).

In S1403, the CPU 261 corrects all of the pixels positioned within the contour selected in S973 to be the estimated background pixels. If the contour 1307 is selected in S973, all estimated handwriting pixels within the contour 1307 are corrected to estimated background pixels as shown in FIG. 15(c). After that, the process transitions to S973.
In S1404, the CPU 261 corrects all the pixels positioned within the contour selected in S973 to be handwriting presumed pixels. When the contour 1309 is selected in S973, all estimated background pixels within the contour 1309 are corrected to estimated handwritten pixels as shown in FIG. 15(c). After that, the process transitions to S973.

According to the above-described correction processing, the image processing server 103 calculates the number of pixels estimated to be handwritten and the number of pixels estimated to be background among the pixels located within the contour existing in the processing target area. are compared, and the estimation result is corrected according to the result of the comparison. Specifically, if the number of pixels estimated to be handwritten is greater, the estimation result is corrected so that the entirety of the outline indicates the handwritten portion. If the number of pixels estimated to be handwritten is smaller, the estimation result is corrected so that the entirety of the contour indicates the background portion.

According to the second embodiment as described above, after estimating a handwritten portion, the number of pixels estimated to be handwritten and the number of pixels estimated to be background in the strokes in the processing target area are compared, and the result of the comparison is Estimate errors can be corrected accordingly. Specifically, the estimation result at the position of the stroke is combined with the number of pixels estimated for the handwriting and the number of pixels estimated for the background, whichever has the larger number of pixels. can be corrected. As a result, it is possible to prevent a part of the character from being lost or an extra part from being added to the handwritten image to be processed. That is, it is possible to improve the estimation accuracy of the handwritten character area.

As a first modified example of this embodiment, the process of S975 may be skipped. In this case, the image processing server 103 does not screen the extracted contours according to the shape of the circumscribing rectangle of the contours. This area is to be corrected.

As described above, the present invention has been described together with the embodiments, but the above-described embodiments merely show specific examples for implementing the present invention, and the technical scope of the present invention should not be construed in a limited manner. It should not be. That is, the present invention can be embodied in various forms without departing from its technical concept or main features.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The present invention may be applied to a system composed of a plurality of devices or to an apparatus composed of one device. For example, although the image processing apparatus 101 and the image processing server 103 have been described as separate apparatuses, the image processing server 103 may have the functions of the image processing apparatus 101 . Further, although the image processing server 103, the printed character OCR server 104, and the handwritten OCR server 105 have been described as separate devices, the image processing server 103 may have the functions of the printed character OCR server 104 and the handwritten OCR server 105. good. Further, although the printed character OCR server 104 and the handwritten OCR server 105 have been described as separate devices, the printed character OCR server 104 and the handwritten OCR server 105 may be configured integrally. Also, although the image processing server 103 and the learning device 102 have been described as separate devices, the image processing server 103 may have the function of the learning device 102 .

100: Image processing system, 101: Image processing device, 102: Learning device, 103: Image processing server, 104: Type OCR server, 105: Handwriting OCR server

Claims (20)

Acquisition means for acquiring a read image of a document including handwriting;
a first extraction means for extracting a processing target area in the read image;
estimating means for estimating a handwritten portion and a background portion in the read image;
a second extraction means for extracting a contour existing in the processing target area in the read image;
control means for controlling correction of the estimation result by the estimation means based on the coordinate position of the extracted contour, the estimated coordinate position of the handwritten portion, and the estimated coordinate position of the background portion;
An image processing device comprising: When the circumscribing rectangle of the extracted contour satisfies a predetermined condition and the contour region includes the handwritten portion and the background portion, the control means controls the contour region. 2. The image processing apparatus according to claim 1, wherein the estimation result is corrected so that the entire inside becomes the background portion. 2. The control means corrects the estimation result based on the number of pixels in the handwritten portion and the number of pixels in the background portion among the pixels in the contour area. The image processing device according to . When the number of pixels in the handwritten portion is greater than the number of pixels in the background portion, the control means corrects the estimation result so that the entire contour area becomes the handwritten portion. 4. The image processing apparatus according to claim 3. When the number of pixels in the handwritten portion is smaller than the number of pixels in the background portion, the control means corrects the estimation result so that the entire contour area becomes the background portion. 5. The image processing apparatus according to claim 3 or 4. The control means controls to correct the estimation result when the circumscribing rectangle of the extracted contour satisfies a predetermined condition, and controls the estimation result when the circumscribing rectangle does not satisfy the predetermined condition. 6. The image processing apparatus according to any one of claims 3 to 5, wherein control is performed so as not to correct . 7. The image processing apparatus according to claim 2, wherein said predetermined condition is a condition indicating a circumscribed rectangle of an outline of a predetermined type. 8. The image processing apparatus according to claim 7, wherein said predetermined characters are brackets or hyphens. The control means corrects the estimation result when the handwritten portion does not exist around the outside of the contour even when the handwritten portion and the background portion are included in the area of the contour. 3. The image processing apparatus according to claim 2, wherein control is performed so as not to. wherein, when a plurality of contours are extracted from the processing target area, the control means sequentially determines whether or not to correct the estimation result for the plurality of extracted contours. Item 10. The image processing apparatus according to any one of Items 1 to 9. 11. The image processing apparatus according to any one of claims 1 to 10, wherein a handwritten image generated based on the estimation result corrected by the control means is subjected to OCR corresponding to handwritten characters. . 12. The image processing apparatus according to any one of claims 1 to 11, wherein a background image generated based on the estimation result corrected by the control means is subjected to OCR corresponding to printed characters. 13. The image processing apparatus according to any one of claims 1 to 12, wherein the estimating means estimates pixels of the handwritten portion in the read image by inputting the read image into a neural network. . An image obtained by synthesizing a first read image obtained by reading a document containing only handwriting and a second read image obtained by reading a document containing only objects other than handwriting is used as an input image. 14. The image processing apparatus according to claim 13, further comprising learning means for learning said neural network using learning data in which pixels are correct data. generating the input image by synthesizing the images after performing predetermined image processing on at least one of the first read image and the second read image; 15. The image processing apparatus according to claim 14, characterized by: 16. The image processing apparatus according to claim 15, wherein the predetermined image processing includes at least one of rotation, scaling, brightness change, and image clipping. 17. The image processing apparatus according to any one of claims 1 to 16, wherein the document is a form. An image processing system including an image generation device that generates a read image of a document including handwriting, an image processing device, and an OCR device,
The image processing device is
acquisition means for acquiring the read image from the image generation device;
a first extraction means for extracting a processing target area in the read image;
estimating means for estimating a handwritten portion and a background portion in the read image;
a second extraction means for extracting a contour existing in the processing target area in the read image;
control means for controlling correction of the estimation result by the estimation means based on the coordinate position of the extracted contour, the estimated coordinate position of the handwritten portion, and the estimated coordinate position of the background portion;
A handwritten image generated based on the estimation result corrected by the control means is subjected to handwritten OCR, and a background image generated based on the estimation result corrected by the control means is subjected to printed character OCR. a transmitting means for transmitting to the device;
has
The OCR device is
receiving means for receiving the handwritten OCR target and the printed character OCR target from the image processing apparatus;
a processing means for performing OCR corresponding to handwritten characters on the handwritten OCR target, and performing OCR corresponding to printed characters on the printed character OCR target;
An image processing system comprising: an acquisition step of acquiring a read image of a document including handwriting;
a first extraction step of extracting a processing target area in the read image;
an estimation step of estimating a handwritten portion and a background portion in the read image;
a second extraction step of extracting a contour existing in the processing target area in the read image;
a control step of controlling to correct the estimation result obtained by the estimation step based on the coordinate position of the extracted contour, the estimated coordinate position of the handwritten part, and the estimated coordinate position of the background part;
An image processing method comprising: A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 17.

Images