JP2020149288A - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

To improve the accuracy in character recognition for a voucher image.SOLUTION: An image processing apparatus comprises an extraction unit, and a processing unit, and performs character recognition for a voucher image showing a voucher. The extraction unit extracts, from the voucher image, a plurality of provisional character areas arranged side by side in a row direction. The processing unit sets a plurality of character interval candidates as an interval for partitioning a row area including the plurality of provisional character areas in the row direction, calculates, for each of the character interval candidates, the degree of overlapping between partitioning positions and a character present portion in accordance with the character interval candidates, determines a character interval from the plurality of character interval candidates based on the plurality of calculated degrees of overlapping, and recognizes the characters for partitioned areas obtained by partitioning the row area according to the determined character interval.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image processing device, an image processing method, and an image processing program that perform character recognition on a voucher image representing a voucher.

OCR recognizes a plurality of characters contained in an image obtained by scanning a voucher such as a receipt or a receipt with a scanner. Here, OCR is an abbreviation for Optical Character Reader as a device, and is an abbreviation for Optical Character Recognition in the sense of optical character recognition. In addition to the recognized characters, the OCR as a device also outputs information representing a character area in the image.

Patent Document 1 discloses a check processing device that recognizes characters printed on the surface of a check. This check processing device performs character recognition on the first cutout area cut out from the image data output from the scanner, outputs the first candidate, and performs character recognition on the second cutout area cut out from the above-mentioned image data. And output the second candidate. The size of the second cutout area is the same as the size of the first cutout area.

Japanese Unexamined Patent Publication No. 2006-127375

OCR may cut out a plurality of characters as one area from an image, or cut out an area less than one character as one character. Since the size of the cutout area does not change, the check processing device described above cannot recognize a plurality of characters cut out as one area or characters covering an area less than one character.

The image processing device of the present invention is an image processing device that recognizes characters on a voucher image representing a voucher.
An extraction unit that extracts a plurality of provisional character areas arranged in the line direction from the voucher image,
With a processing unit,
The processing unit
A plurality of character spacing candidates are set as intervals for partitioning the line area including the plurality of provisional character areas in the line direction.
For each of the character spacing candidates, the degree of overlap between the division position according to the character spacing candidate and the character existence portion is calculated.
Based on the calculated multiplicity, the character spacing is determined from the plurality of character spacing candidates.
It has an embodiment in which characters are recognized for a partition area in which the line area is partitioned according to the determined character spacing.

Moreover, the image processing method of the present invention
It is an image processing method that recognizes characters on a voucher image that represents a voucher.
An extraction step of extracting a plurality of provisional character areas arranged in the line direction from the voucher image, and
Including the processing process,
In the processing step,
A plurality of character spacing candidates are set as intervals for partitioning the line area including the plurality of provisional character areas in the line direction.
For each of the character spacing candidates, the degree of overlap between the division position according to the character spacing candidate and the character existence portion is calculated.
Based on the calculated multiplicity, the character spacing is determined from the plurality of character spacing candidates.
It has an aspect including a processing step of recognizing characters with respect to a partition area in which the line area is partitioned according to the determined character spacing.

Further, the image processing program of the present invention
An image processing program for character recognition of a voucher image representing a voucher.
An extraction function that extracts a plurality of provisional character areas arranged in the line direction from the voucher image, and
Realize the processing function on the computer,
The processing function
A plurality of character spacing candidates are set as intervals for partitioning the line area including the plurality of provisional character areas in the line direction.
For each of the character spacing candidates, the degree of overlap between the division position according to the character spacing candidate and the character existence portion is calculated.
Based on the calculated multiplicity, the character spacing is determined from the plurality of character spacing candidates.
It has an embodiment in which characters are recognized for a partition area in which the line area is partitioned according to the determined character spacing.

A block diagram schematically showing a configuration example of a system including an image processing device. A flowchart showing an example of character recognition processing. The flowchart which shows the example of the character spacing determination processing. The figure which shows typically the example of the recognition result by an OCR engine. The figure which shows each component of a typeface schematically. The figure which shows typically the example which determines the character spacing. The figure which shows typically the example of the degree of overlap of the section position and the character existence part. The figure which shows typically the example of changing a part of the half-width character spacing to the full-width character spacing. The figure which shows typically the example which character recognition was performed by OCR on the voucher image.

Hereinafter, embodiments of the present invention will be described. Of course, the following embodiments merely exemplify the present invention, and not all of the features shown in the embodiments are essential for the means for solving the invention.

(1) Outline of the technique included in the present invention:
First, an outline of the technique included in the present invention will be described with reference to the examples shown in FIGS. 1 to 9. The figures of the present application are diagrams schematically showing examples, and the enlargement ratios in each direction shown in these figures may be different, and the figures may not be consistent. Of course, each element of the present technology is not limited to the specific example indicated by the reference numeral. In "Overview of the technique included in the present invention", the parentheses mean a supplementary explanation of the immediately preceding word.

Aspect 1:
As illustrated in FIG. 1 and the like, the image processing device (for example, the main server 30) according to one aspect of the present technology is an image processing device (30) that recognizes characters on the voucher image IM1 representing the voucher. It includes an extraction unit U1 and a processing unit U2. The extraction unit U1 extracts a plurality of provisional character regions A1 arranged in the line direction D1 from the voucher image IM1. The processing unit U2 sets a plurality of character spacing candidates i as intervals for partitioning the line area A10 including the plurality of provisional character areas A1 in the line direction D1, and for each of the character spacing candidates i, the characters. The degree of overlap between the division position P1 and the character presence portion A1 according to the interval candidate i (for example, the ratio Ri of the character non-existing pixels) is calculated, and the plurality of the character interval candidates i are based on the calculated degree of overlap (Ri). Character spacing (for example, widths W3 and W4) is determined from the above, and characters are recognized with respect to the partition area A2 that divides the line area A10 according to the determined character spacing (W3, W4).

In the above-described aspect 1, the line area A10 is set based on the degree of overlap (Ri) between the partition position P1 according to each character spacing candidate i and the character existence portion A11 in which a plurality of characters exist in the line area A10. The character spacing (W3, W4) for partitioning is determined, and characters are recognized for the partitioning area A2 for partitioning the line area A10 according to the character spacing (W3, W4). Since the space between characters used for vouchers is often limited to a small number of types, the influence of faintness on the cutting out of each character is reduced. Therefore, this aspect can improve the accuracy of character recognition for the voucher image.

Here, the voucher means an accounting material obtained from an external third party, and includes a receipt, a receipt, an invoice, a delivery note, and the like.
Since the non-existing part of the character in the line area excluding the existing part of the character, the degree of overlap between the division position and the existing part of the character is the ratio of the overlapping character existence pixel to the multiple division positions. Includes the percentage of non-existent characters that overlap with multiple partition positions, etc.
The character spacing is not limited to equal spacing, and a plurality of spacing may be mixed, such as a mixture of half-width character spacing and full-width character spacing. The character spacing candidates are not limited to equal spacing, and a plurality of spacing may be mixed, such as a mixture of half-width character spacing and full-width character spacing.
The above-mentioned additional notes also apply in the following aspects.

Aspect 2:
As illustrated in FIG. 4, the extraction unit U1 may associate the arrangement information IA1 representing the arrangement of the provisional character area A1 with the provisional character area A1. The processing unit U2 may determine the height H0 of the line area A10 based on the arrangement information IA1 associated with each of the provisional character areas A1, and the plurality of heights H0 based on the determined height H0. Character spacing candidate i may be set. Since the characters used for the voucher are often spaced according to the height of the line, this aspect can speed up the process.

Aspect 3:
As illustrated in FIGS. 3 and 6, the processing unit U2 sets a half-width character interval (for example, width W3) that temporarily divides the line area A10 in half-width character units based on the degree of overlap (Ri). May be good. As illustrated in FIG. 8 and the like, the processing unit U2 overlaps with the character existence portion A11 among a plurality of half-width division positions P2 according to the set half-width character spacing (W3) (for example, of the character existence pixel). The half-width character spacing (W3) may be changed to the full-width character spacing (for example, width W4) by excluding the half-width division position where the number Nj) is larger than the reference degree (for example, the threshold value Tn) from the division of the line area A10. .. Further, the processing unit U2 recognizes characters for each partition area A2 according to the character spacing (W3, W4) which is one of the half-width character spacing (W3) and the full-width character spacing (W4). You may. Since half-width characters and full-width characters are often used for the voucher, this aspect can further improve the accuracy of character recognition for the voucher image.
Here, since the non-existing character portion is the portion excluding the existing character portion in the line area, the degree to which the half-width division position overlaps with the existing character portion is the number of characters existing pixels overlapping with the half-width division position and the half-width Includes the number of non-existing characters pixels that overlap with the partition position, etc. This appendix also applies in the following aspects:

Aspect 4:
The processing unit U2 uses a character recognition model (for example, the DL engine 32 shown in FIG. 1) generated by machine learning for recognizing characters included in an image representing a voucher to display characters for each of the compartment areas A2. May be recognized. This aspect can provide a suitable example of improving the accuracy of character recognition for a voucher image.

Aspect 5:
As illustrated in FIG. 2, the extraction unit U1 may extract the plurality of provisional character regions A1 from the voucher image IM1 by performing the first character recognition on the voucher image IM1. The processing unit U2 may perform a second character recognition with higher accuracy than the first character recognition for each of the compartment areas A2. This aspect can also provide a suitable example of improving the accuracy of character recognition for a voucher image.

Aspect 6:
Further, as illustrated in FIG. 2 and the like, the image processing method according to one aspect of the present technology includes an extraction step ST1 corresponding to the extraction unit U1 and a processing step ST2 corresponding to the processing unit U2. This aspect can also improve the accuracy of character recognition for the voucher image.

Aspect 7:
Further, as illustrated in FIG. 1, the image processing program PR1 according to one aspect of the present technology uses a computer (for example, a main server) for the extraction function FU1 corresponding to the extraction unit U1 and the processing function FU2 corresponding to the processing unit U2. Realize in 30). This aspect can also improve the accuracy of character recognition for the voucher image.

Further, the present technology can read a composite device including the above-mentioned image processing device, an information processing method including the above-mentioned image processing method, an information processing program including the above-mentioned image processing program, and a computer-readable record of any of the above-mentioned programs. It can be applied to various media, etc. Any of the devices described above may be composed of a plurality of dispersed parts.

(2) Background to the idea of this technology:
FIG. 9 schematically shows an example in which character recognition is performed by OCR on the voucher image IM1 obtained by scanning the receipt which is the voucher with a scanner. A receipt means an unaddressed receipt that is mechanically issued at a cash register. The OCR extracts a plurality of recognized character areas A5 including the recognized characters from the voucher image IM1.

For example, in the voucher image IM1, the character "\" in the upper line is correctly recognized as one character. In this case, the recognition character area A5 including only one character is extracted from the voucher image IM1, and the correctly recognized characters and the information representing the range of the recognition character area A5 are output from the OCR. The information representing the range of the recognition character area A5 will be described in detail later, but includes information indicating the start coordinates of the recognition character area A5, information indicating the width of the recognition character area A5, information indicating the height of the recognition character area A5, and the like. .. On the other hand, in FIG. 9, the characters "0" and "6" having the faint PA1 are recognized as two characters, respectively, and are not correctly recognized. In FIG. 9, the character "0" containing the PA1 is recognized as two characters "(" and ")", and the linear blur facing the height direction D2. It is shown that the letter "6" containing PA1 is recognized as the two letters "E" and "j". In particular, when thermal paper is used for the voucher, a part of the character is easily blurred, and therefore the character may not be recognized correctly. Since thermal paper is often used for receipts issued by cash registers, characters are likely to be faint.

In addition, dirt may adhere to the receipt. In FIG. 9, the two characters "20", the two characters "8 years", and the two characters "23", which are close to the dirty SP1, are each recognized as one character and are not correctly recognized. In particular, the width of the recognition character area A5 for the two characters "8 years" is the total width of the half-width characters and the full-width characters, and does not match the width of the characters used in the receipt.

In receipts issued by cash registers, the width of full-width characters is constant, and the width of half-width characters is often half the width of full-width characters. Therefore, in the present technology, the character spacing for partitioning the line area A10 surrounding a plurality of characters arranged in the line direction D1 in the line direction D1 is selected from a plurality of candidates, and the characters in each character area are selected according to the selected character spacing. By recognizing, the accuracy of character recognition is improved. A specific example of this technique will be described below.

(3) Specific example of system configuration including image processing device:
FIG. 1 schematically shows a specific example of the configuration of a system including an image processing device. The system SY1 shown in FIG. 1 includes a scanner 10, a client 20, a main server 30 which is an example of an image processing device, and a storage server 40. Here, the client is an abbreviation for a client computer, the main server is an abbreviation for a main server computer, and a storage server is an abbreviation for a storage server computer. The main server 30 is an example of an image processing device. The client 20, the main server 30, and the storage server 40 are connected to the network NE1 including the Internet. The connection to the network NE1 may be a wired connection, a wireless connection, or both a wired and wireless connection. The network NE1 including the Internet may include a LAN. Here, LAN is an abbreviation for Local Area Network. The main server 30 and the storage server 40 can provide cloud services to users via the network NE1.

For example, the scanner 10 irradiates the original with light from a light source to read the original image, and outputs the original image as data to the outside. The scanner 10 shown in FIG. 1 is connected to the communication I / F 20h of the client 20 so as to be able to communicate by wire or wirelessly. Here, I / F is an abbreviation for an interface. When the scanner 10 optically reads the voucher as a document, the scanner 10 outputs the corresponding voucher image IM1 to the client 20. As the scanner 10, a flatbed type scanner in which a document is arranged between the platen glass and the document cover, a scanner with a document feeder, and the like can be used. Further, the scanner 10 may be a multifunction device having functions such as a printing function and a facsimile communication function in addition to the scanner function.

As the client 20, a personal computer including a tablet terminal, a smartphone, or the like can be used. The client 20 shown in FIG. 1 includes a CPU 20a as a processor, a ROM 20b as a semiconductor memory, a RAM 20c as a semiconductor memory, a storage device 20d, a client network I / F20e, an input device 20f, a display device 20g, a communication I / F20h, and the like. have. Since these elements 20a to 20h and the like are electrically connected, information can be input and output from each other. Here, CPU is an abbreviation for Central Processing Unit, ROM is an abbreviation for Read Only Memory, and RAM is an abbreviation for Random Access Memory.

Although not shown, the storage device 20d stores an operating system, an application program, and the like. The application program includes a driver program that controls the scanner 10. This driver program provides the client 20 with a function of receiving the voucher image IM1 generated by the scanner 10 reading the voucher from the scanner 10 and a mechanism of uploading the voucher image IM1 to the main server 30. As the storage device 20d, a magnetic storage device such as a hard disk, a non-volatile semiconductor memory such as a flash memory, or the like can be used. The CPU 20a appropriately reads the information stored in the storage device 20d into the RAM 20c, and executes various processes by executing the read program. The network I / F20e is connected to the network NE1 and communicates with a partner device connected to the network NE1 according to a predetermined communication standard. As the input device 20f, a pointing device, a hard key including a keyboard, a touch panel attached to the surface of a display panel, and the like can be used. A liquid crystal display panel or the like can be used for the display device 20 g. The communication I / F 20h is an interface for connecting to a peripheral device such as a scanner 10. A USB interface, a wireless communication interface, or the like can be used for the communication I / F 20h. Here, USB is an abbreviation for Universal Serial Bus.

The scanner 10 may include the configuration of the client 20. In this case, the scanner 10 functions as a device having a function of communicating with the outside via the network NE1.

The main server 30 is a server computer that provides a character recognition function on the network NE1, and may be one computer or a plurality of computers. The main server 30 shown in FIG. 1 includes a CPU 30a as a processor, a ROM 30b as a semiconductor memory, a RAM 30c as a semiconductor memory, a storage device 30d, a network I / F30e for the main server, and the like. Since these elements 30a to 30e and the like are electrically connected, information can be input and output from each other. The processor is not limited to one CPU, but may be a plurality of CPUs, a combination of a hardware circuit such as an ASIC and a CPU, and the like. Here, ASIC is an abbreviation for Application Specific Integrated Circuit. Although not shown, the main server 30 may be provided with an input device for receiving an operation by the operator, a display device for showing information to the operator, and the like.

The storage device 30d stores an operating system (not shown), an image processing program PR1, and the like. As the storage device 30d, a magnetic storage device such as a hard disk, a non-volatile semiconductor memory such as a flash memory, or the like can be used. The network I / F30e is connected to the network NE1 and communicates with a partner device connected to the network NE1 according to a predetermined communication standard.

The CPU 30a shown in FIG. 1 includes a general-purpose OCR engine 31 which is an example of the extraction unit U1 and a DL engine 32 which is an example of a character recognition unit using AI. Here, AI is an abbreviation for Artificial Intelligence, and DL is an abbreviation for Deep Learning. The character recognition unit is a part of the processing unit U2. The OCR engine 31 may be an extraction function FU1 realized by the image processing program PR1 on the main server 30. The DL engine 32 may be a character recognition function realized by the image processing program PR1 on the main server 30. The character recognition function is a part of the processing function FU2.
The image processing program PR1 shown in FIG. 1 includes a control program 33 that realizes a control function on the main server 30. The control function is a part of the processing function FU2.

The CPU 30a of the main server 30 appropriately reads the information stored in the storage device 30d into the RAM 30c and executes various processes by executing the read program. The CPU 30a performs processing corresponding to the above-mentioned function by executing the image processing program PR1 read into the RAM 30c. When the image processing program PR1 realizes the above-mentioned functions FU1 and FU2 on a computer, the image processing program PR1 corresponds to the main server 30, which is a computer, the extraction unit U1 corresponding to the extraction function FU1 and the processing function FU2. It functions as a processing unit U2. Further, the main server 30 that executes the image processing program PR1 executes the extraction process ST1 corresponding to the extraction unit U1 and the processing process ST2 corresponding to the processing unit U2. The computer-readable medium that stores the image processing program PR1 that realizes the above-mentioned functions FU1 and FU2 in a computer is not limited to the storage device inside the main server 30, and may be a recording medium outside the main server 30.

The main server 30 shown in FIG. 1 is communicably connected to the storage server 40. The storage server 40 is a server computer that provides a storage function on the network NE1, and may be one computer or a plurality of computers. Although not shown, the storage server 40 includes a CPU as a processor, ROMs and RAMs as semiconductor memories, an internal storage device, a network I / F for a storage server, and the like. The storage server 40 can store the data received from the main server 30 in the internal storage device, and can transmit the data stored in the internal storage device via the network NE1.
Of course, the fact that the server computer is divided into the main server 30 and the storage server 40 is only an example, and the main server 30 may include the configuration of the storage server 40.

(4) Specific example of character recognition processing:
FIG. 2 schematically illustrates the character recognition process performed by the main server 30. FIG. 3 schematically illustrates the character spacing determination process performed in step S110 of FIG. Here, steps S102 to S104 of FIG. 2 correspond to the extraction unit U1, the extraction step ST1, and the extraction function FU1. Steps S106 to S112 of FIG. 2 and steps S202 to S212 of FIG. 3 correspond to the processing step ST2. Hereinafter, the description of "step" will be omitted. Further, it is described that what the extraction unit U1 does is performed by the OCR engine 31, and what the character recognition unit using AI performs is described as being performed by the DL engine 32.

When the user causes the scanner 10 to read a voucher such as a receipt, the scanner 10 generates a voucher image IM1 representing the read voucher and transmits the voucher image IM1 to the client 20. The client 20 receives the voucher image IM1 and transmits it to the main server 30 via the network NE1. Therefore, the main server 30 receives the voucher image IM1 via the network NE1 in S102. An example of the voucher image IM1 is shown in FIG.

The main server 30 that has acquired the voucher image IM1 causes a general-purpose OCR engine 31 to execute the first character recognition process for the voucher image IM1 in S104. As illustrated in FIG. 9, the OCR engine 31 extracts a plurality of recognized character areas A5 including arrangement information from the voucher image IM1 by performing the first character recognition on the voucher image IM1. The plurality of recognized character areas A5 include a plurality of provisional character areas A1 arranged in the line direction D1. In this specific example, a plurality of provisional character areas A1 arranged in the line direction D1 will be described.

The OCR engine 31 analyzes the layout of the voucher image IM1, cuts out a line based on the analysis result, and for the one-character-like image included in each cut-out line, creates a provisional character area A1 surrounding the one-character-like image. Set. In FIG. 9, "¥", both sides of "0" having a faint PA1, both sides of "6" having a faint PA1, "20" close to dirt, "8 years" close to dirt, "23" close to dirt, It is shown that the provisional character area A1 of one character is set in the above. Next, the OCR engine 31 estimates the characters in the provisional character area A1 from the image of each provisional character area A1 according to a predetermined algorithm. In the provisional character area A1 that does not correspond to one character, the correct character is not recognized in many cases. The OCR engine 31 outputs the recognition character together with the arrangement information indicating the arrangement of each provisional character area A1.

FIG. 4 schematically illustrates the recognition result RE9 output from the OCR engine 31. At the bottom of FIG. 4, a schematic provisional character area A1 for explaining each item of the arrangement information IA1 representing the arrangement of the provisional character area A1 is shown. The recognition result RE9 includes the recognition character and the arrangement information IA1 associated with each provisional character area A1. The arrangement information IA1 represents, for example, the arrangement of the provisional character area A1 by a numerical value in units of pixels in the voucher image IM1. The layout information IA1 shown in FIG. 4 includes a character left end Xs indicating the position of the left end of the provisional character area A1, a character upper end Ys indicating the position of the upper end of the provisional character area A1, and a character width Wc indicating the width of the provisional character area A1. A character height Hc representing the height of the provisional character area A1 and a baseline BL of the provisional character area A1 are included. The left end Xs of the character and the upper end Ys of the character are the starting coordinates of the provisional character area A1, and are represented by, for example, the coordinate values of the pixels in the upper left corner of the provisional character area A1. The character width Wc is represented by, for example, the number of pixels in the provisional character region A1 in the line direction D1. The character height Hc is represented by, for example, the number of pixels in the provisional character region A1 in the height direction D2.

FIG. 5 schematically shows each component of the typeface. A lowercase European typeface is shown in the upper part of FIG. 5, and an uppercase European typeface is shown in the lower part of FIG.

The height of the lowercase letters of the alphabet can be divided into x-height, ascender, and Descender. Also, the virtual line between the X-height and the descender is the Baseline BL, the virtual line at the bottom of the descender is the Descender line DL, and the virtual line between the X-height and the ascender is. It is the Mean line, and the virtual line at the top of the ascender line is the Ascender line AL. The lowercase letters of the alphabet include letters placed between the baseline BL and the meanline ML, such as "x", "a", "c", etc., and "b", "d", etc. There are characters arranged between the baseline BL and the ascender line AL, and characters arranged between the descender line DL and the mean line ML such as "g", "j", and the like.

The height of the uppercase letters of the alphabet is adjusted to the cap height. The virtual line at the lower end of the cap height is the baseline BL, and the virtual line at the upper end of the cap height is the cap line CL. The cap line CL may be designed lower than the ascender line AL or higher than the ascender line AL.

As described above, in the European typeface, a plurality of characters are arranged with the baseline BL aligned. In the case of Japanese as well, a plurality of characters are arranged with reference to the baseline BL.
Therefore, the OCR engine 31 acquires the baseline BL together with the character left end Xs, the character upper end Ys, the character width Wc, and the character height Hc at the time of extracting each provisional character area A1. The OCR engine 31 associates these arrangement information IA1 with the provisional character area A1 together with the recognition character, and outputs the obtained recognition result RE9.

After the first character recognition, the main server 30 acquires the recognition result RE9 output from the OCR engine 31, that is, the recognition character and the arrangement information IA1 associated with each provisional character area A1 in S106.

After that, in S108, the main server 30 sets the line area A10 including the plurality of provisional character areas A1 arranged in one line based on the arrangement information IA1 of each provisional character area A1 arranged in the line direction D1. Set. As shown in FIG. 5, since the baseline BL is aligned with the plurality of characters on the same line, the plurality of provisional character areas A1 having substantially the same baseline BL are arranged on the same line. Here, let Yb be the position of the baseline BL of the provisional character region A1 that was first noticed, and let Tb be a minute threshold value with respect to the baseline BL. The main server 30 may set the smallest rectangle including a plurality of provisional character areas A1 in which the position of the baseline BL is in the range from (Yb-Tb) to (Yb + Tb) as the line area A10. In the example shown in FIG. 4, since a plurality of provisional character areas A1 having a baseline BL of "103" are arranged on the same line, the smallest rectangle including these provisional character areas A1 is the line area A10. Is set as.

In the middle of FIG. 9, it is shown that a line area A10 including a plurality of provisional character areas A1 is set for each line. For example, the main server 30 searches for the leftmost coordinate value XL, the rightmost coordinate value XR, the top coordinate value YU, and the lowest coordinate value YD in a plurality of provisional character areas A1 for one line. And, the rectangle from the upper left coordinate (XL, YU) to the lower right coordinate (XR, YD) can be set as the row area A10. Since the height from the lowest coordinate value YD to the highest coordinate value YU corresponds to the height of the row area A10, S108 indicates a process of determining the height of the row area A10. When the coordinate values YU and YD are in pixel units, the height of the row area A10 is | YD-YU | -1.

FIG. 6 schematically illustrates how to determine the character spacing of the widths W3 and W4 from the set line area A10. The uppermost part of FIG. 6 corresponds to S108 and shows a state in which the height H0 of the row area A10 indicated by the virtual line is determined. Hereinafter, description will be made with reference to FIG.

After determining the height H0 of the line area A10, the main server 30 performs a process of determining the character spacing of each line area A10 in S110. FIG. 3 shows the character spacing determination process performed in S110.

First, in S202, the main server 30 sets the width W1 which is half the height H0 of the line area A10 to a temporary half-width character spacing. FIG. 6 shows how the line area A10 is divided in the line direction D1 at temporary half-width character intervals of width W1 = H0 / 2.

After setting the temporary half-width character spacing, the main server 30 sets, in S204, a plurality of character spacing candidates i for partitioning the line area A10 in the line direction D1 based on the temporary half-width character spacing of the width W1. “I” is a variable that identifies a character spacing candidate. Here, the width of the character spacing of the character spacing candidate i is W2 (i), and the amount of change in the character spacing centered on the temporary half-width character spacing of the width W1 is ΔW. In this case, the main server 30 sets the character spacing width W2 (i) of the character spacing candidate i in a plurality of stages from W1-ΔW to W1 + ΔW.
From the above, S202 to S204 indicate a process of setting a plurality of character spacing candidates i based on the height H0 of the line area A10.

After setting the plurality of character spacing candidates i, the main server 30 acquires, in S206, the ratio Ri of the non-existing character pixels at the section positions according to the set character spacing candidates i for each character spacing candidate i.

FIG. 7 schematically illustrates the ratio Ri of the non-existing characters of the character at the division position P1 according to the character spacing candidate i. The resolution of the voucher image IM1 shown in FIG. 7 is low for easy understanding, and the resolution of the actual voucher image IM1 is assumed to be higher than the resolution shown in FIG.
The voucher image IM1 includes a pixel of a character nonexistent portion A12 having a pixel value representing a ground color such as white, and a pixel of a character present portion A11 having a pixel value representing a print color such as black. Whether each pixel included in the voucher image IM1 is the character presence portion A11 or the character non-existence portion A12 depends on whether the pixel value is the value on the ground color side or the value on the print color side based on the threshold value for the pixel value. It can be determined by examining whether or not it exists. In the voucher image IM1 shown in FIG. 7, the black or gray pixel is the character presence portion A11, and the white pixel is the character non-existence portion A12. Here, the ratio Ri of the character non-existing pixels represents the degree of overlap between the division position P1 and the character existing portion A11.

FIG. 7 shows each section position P1 when the character spacing candidate i is set to a spacing of 4 to 6 pixels. In FIG. 7, the pixels at the plurality of partition positions P1 are marked with a cross. Note that the character spacing candidate i having a 4-pixel spacing means that there are 3 pixels between adjacent partition positions P1 in the row direction D1, and the character spacing candidate i having a character spacing candidate i having a 6-pixel spacing means that the character spacing candidate i has a row direction D1. It means that there are 5 pixels between adjacent partition positions P1. In the example shown in FIG. 7, when the character spacing candidate i is a 4-pixel spacing, the number of pixels of the total partition position P1 is 80, and the number of pixels of the character nonexistent portion A12 at the total partition position P1 is 63. The ratio Ri of the character non-existing portion is 0.788 (= 63/80). Further, when the character spacing candidate i is 5 pixel spacing, the number of pixels of the total partition position P1 is 64, the number of pixels of the character nonexistent portion A12 at the total partition position P1 is 54, and the character nonexistent portion The ratio Ri is 0.844 (= 54/64). Further, when the character spacing candidate i has a 6-pixel spacing, the number of pixels of the total partition position P1 is 56, the number of pixels of the character nonexistent portion A12 at the total partition position P1 is 35, and the number of pixels of the character nonexistent portion A12 is 35. The ratio Ri is 0.625 (= 35/56).

After acquiring the ratio Ri of the non-existing characters, the main server 30 determines in S208 the width W2 (i) of the character spacing candidate i having the maximum ratio Ri as the half-width character spacing of the width W3. The width W3 is a determined width of half-width characters. FIG. 6 shows how the half-width character spacing of the width W3 is selected from the character spacing of the plurality of widths W2 (i). In the example shown in FIG. 7, since the ratio Ri when the character spacing candidate i is 5 pixel spacing is the maximum, the half-width character spacing is determined to be 5 pixel spacing.
S206 to S208 described above show a process of setting a half-width character interval having a width W3 that temporarily divides the line area A10 in half-width character units based on the ratio Ri of non-existing characters.

In the above-mentioned example, in the voucher image, the process of determining the character spacing candidate in 1-pixel units using the spacing of 4, 5, and 6 pixels as an example has been described, but the same applies to the voucher image having a higher resolution and a higher resolution image. May be performed. In this way, the accuracy of obtaining the character spacing can be further improved.

After setting the half-width character spacing, the main server 30 acquires the number of character existence pixels at each half-width division position j according to the half-width character spacing of the width W3 in S210.

FIG. 8 schematically illustrates how a part of the half-width character spacing is changed to the full-width character spacing. Here, let j be a variable that identifies each of the plurality of half-width section positions P2, and let Nj be the number of pixels of the character existence portion A11 at the half-width section position j. The number of pixels Nj represents the degree to which the half-width partition position P2 and the character existence portion A11 overlap. In the example shown in FIG. 8, the number Nj of pixels in which each half-width section position j and the character existence portion A11 overlap is 0, 0, 0, 0, 7, 0, 0, 3 in order from the left. ..

After acquiring each number of pixels Nj, the main server 30 removes, in S212, the half-width section position where the number of pixels Nj exceeds the threshold value Tn among the plurality of half-width section positions j from the section of the row area A10. After that, the main server 30 ends the character spacing determination process. The threshold value Tn represents the reference degree with respect to the number of pixels Nj. For example, the threshold value Tn can be a number obtained by multiplying the number of pixels included in one half-width partition position j by a ratio larger than 0 and smaller than 1 (for example, 0.25 or the like). In the example shown in FIG. 8, one half-width partition position j includes 8 pixels, and the threshold value Tn is 2 pixels. Of the plurality of half-width section positions j, the half-width section position of Nj = 7 and the half-width section position of Nj = 3 are removed from the section of the row area A10. As a result, the half-width character spacing sandwiching the half-width section position of Nj = 7 is changed to the full-width character spacing, and the half-width character spacing sandwiching the half-width section position of Nj = 3 is changed to the full-width character spacing. At the bottom of FIG. 8, each section position P3 according to the changed character spacing is shown. FIG. 6 shows how the half-width character spacing of the width W3 is changed to the full-width character spacing of the width W4 = 2 × W3. The width W4 is a determined width of full-width characters.

In S210 to S212 described above, the half-width partition position P2 in which the degree of overlap with the character existence portion A11 among the plurality of half-width partition positions P2 according to the half-width character spacing of the width W3 is larger than the reference degree is set from the section of the line area A10. The process of changing the half-width character spacing of the width W3 to the full-width character spacing of the width W4 is shown by removing it. Therefore, in the processing of S206 to S212, the character spacing of the widths W3 and W4 for partitioning the line area A10 is determined based on the degree of overlap between the partition position P1 according to each character spacing candidate i and the character existence portion A11. Indicates processing.

After the character spacing determination process shown in FIG. 3, the main server 30 has, in S112 of FIG. 2, each partition area A2 partitioned from the line area A10 according to the character spacing of one of the width W3 and the width W4. Is set to the setting area A3, and the DL engine 32 is made to execute the second character recognition. Further, the main server 30 causes the DL engine 32 to output a certainty degree C indicating the certainty of the recognized character. The DL engine 32 executes a second character recognition process for recognizing a character in the setting area A3 and a process for outputting a certainty degree C indicating the certainty of the recognized character. As shown in FIG. 8, the second character recognition process is performed on each partition area A2. The certainty degree C is output for each partition area A2. For example, in each partition area A2 shown in FIG. 8, the recognition characters “2” and C = 1.0, the recognition characters “0” and C = 0.9, and the recognition characters “1” and C in the order of the line direction D1. = 1.0, recognition characters "8" and C = 0.8, recognition characters "year" and C = 1.0, recognition characters "1" and C = 1.0, and recognition characters "month" and C It is shown that = 0.8 is output.

The DL engine 32 is also a type of OCR engine for executing character recognition processing. However, the DL engine 32 includes a character recognition model generated by Deep Learning, which is an example of machine learning, and performs a second character recognition with higher accuracy than the first character recognition performed by the OCR engine 31. The DL engine 32 can be generated by using a known machine learning algorithm such as a neural network. For supervised machine learning, input data with a large amount of learning images representing voucher characters as input and output data with characters included in each of these learning images as output can be used. it can. For example, when the relationship between the input data and the output data is input to the multi-layered neural network as teacher data, the DL engine 32 automatically learns the features of a large amount of training images and includes them in the image representing the voucher. Build a character recognition model that recognizes the characters to be used and estimates the certainty of the characters. The certainty C, which indicates the certainty of the recognized character, is, for example, the ratio of the learning image including the character matching the recognized character among the plurality of learning images when there are a plurality of learning images same as the input image. Corresponds to, and means the probability that the recognized character matches the character included in the input image.
From the above, the DL engine 32 includes a character recognition model generated by machine learning for recognizing characters included in an image representing a voucher and estimating the certainty of the characters. The DL engine 32 recognizes a character with respect to the setting area A3 using the character recognition model, and acquires a certainty degree C indicating the certainty of the recognized character from the character recognition model.

Here, the types of characters that appear on the voucher are limited, such as numbers from "0" to "9", characters that represent the amount of money such as "¥" and "yen", and kanji that represent the date and time such as "month" and "day". Has been done. Therefore, the DL engine 32 can recognize characters with higher accuracy than the OCR engine 31.

After the second character recognition, the main server 30 stores the recognized characters of each partition area A2 in the storage server 40 together with at least the voucher image IM1 in S114. The main server 30 together with the recognition character of each partition area A2 and the voucher image IM1, has the certainty degree C of each partition area A2, the left end of the character of each partition area A2, the upper end of the character of each partition area A2, and each partition area. Information such as the character width of A2, the character height of each partition area A2, the baseline of each partition area A2, and the like may also be stored in the storage server 40. Here, a plurality of recognition characters included in the voucher image IM1 will be referred to as recognition character data. For example, the operator who operates the main server 30 can confirm whether or not the recognition character data is correct by displaying information such as the recognition character data stored in the storage server 40 on the display device. In this case, the operator may perform an operation on the main server 30 to correct the recognition character included in the recognition character data.
After the storage process of S114, the main server 30 ends the character recognition process shown in FIG.

The storage server 40 can transmit information such as recognition character data to the outside via the network NE1. The recognition character data stored in the storage server 40 includes a character string indicating the contents of the transaction partner, transaction date, transaction amount, etc. written on the voucher such as a receipt or an invoice. Therefore, the storage server 40 may transmit information such as recognition character data to a terminal used in an accounting office for processing such as accounting processing and tax processing. Further, information such as recognition character data stored in the storage server 40 may be transmitted to the client 20 at the request of the user, or may be transmitted to a printer connected to the network NE1 and then by the printer. It may be printed.

As described above, the character spacing for partitioning the line area A10 is determined based on the degree of overlap between the partition position P1 according to each character spacing candidate i and the character existence portion A11 of the line area A10, and the character spacing is determined. Characters are recognized for each partition area A2 partitioned from the line area A10 according to the above. The character spacing used for vouchers is often limited to a small number, for example, either a fixed half-width character spacing or a fixed full-width character spacing. From this, even if the characters are faint or dirty in the printed voucher, each character is appropriately cut out according to the breaks of a plurality of characters, and the characters in each section area A2 are recognized with high accuracy. To. Therefore, this specific example can improve the accuracy of character recognition for the voucher image.

(5) Modification example:
Various modifications of the present invention can be considered.
In the specific example described above, the main server 30 executes the image processing program PR1, but the image processing program PR1 may be executed by at least one of the client 20 and the scanner 10. For example, when the client 20 executes the image processing program PR1 stored in the storage device 20d, the above-mentioned functions FU1 to FU3 are realized in the client 20, and the client 20 is an example of the image processing device. It is also possible for the main server 30 and the client 20 to cooperate with each other to execute the image processing program PR1. For example, it is conceivable that the main server 30 executes the OCR engine 31 and the DL engine 32, and the client 20 executes the control program 33. Further, the main server 30 may execute the DL engine 32, and the client 20 may execute the OCR engine 31 and the control program 33.

The second character recognition, which is more accurate than the first character recognition, was created by machine learning methods other than deep learning, in addition to being realized by the DL engine including the character recognition model generated by deep learning. It may be realized programmatically.

In the processing of S206 to S208 of FIG. 3, the half-width character spacing is determined based on the ratio Ri of the character non-existing pixels at the division position, but the half-width character spacing may be determined based on the ratio of the character existence pixels at the division position. It is possible. For example, the main server 30 may acquire the ratio 1-Ri of the character existence pixels at the division positions according to each character spacing candidate i in S206, and the character spacing candidate i in which the ratio 1-Ri is the minimum in S208. The width W2 (i) of may be determined as the half-width character spacing of the width W3. In this case, the ratio 1-Ri of the character existence pixels represents the degree of overlap between the division position P1 and the character existence portion A11.

In the processing of S210 to S212 of FIG. 3, the half-width section position to be removed from the section of the line area A10 was determined based on the number of pixels Nj in which the characters exist at each half-width section position j, but the character does not exist at each half-width section position j. It is also possible to determine the half-width section position to be removed from the section of the row area A10 based on the number of pixels. For example, the main server 30 may acquire the number of non-existing character pixels of each half-width partition position j according to the half-width character spacing in S210, and in S212, the number of non-existing character pixels is less than the threshold value. It may be removed from the section of row area A10. In this case, the number of non-existing character pixels represents the degree of overlap between the half-width partition position P2 and the character existing portion A11, and the above-mentioned threshold value represents the reference degree.

(6) Conclusion:
As described above, according to the present invention, it is possible to provide a technique for improving the accuracy of character recognition for a voucher image or the like in various aspects. Of course, the above-mentioned basic actions and effects can be obtained even with a technique consisting of only the constituent requirements according to the independent claims.
In addition, the configurations disclosed in the above-mentioned examples are mutually replaced or the combinations are changed, the known techniques and the respective configurations disclosed in the above-mentioned examples are mutually replaced or the combinations are changed. It is also possible to implement the above-mentioned configuration. The present invention also includes these configurations and the like.

10 ... Scanner, 20 ... Client, 30 ... Main server, 30d ... Storage device, 31 ... OCR engine, 32 ... DL engine, 33 ... Control program, 40 ... Storage server, A1 ... Provisional character area, A2 ... Partition area, A3 ... Setting area, A5 ... Recognition character area, A10 ... Line area, A11 ... Character presence part, A12 ... Character non-existence part, C ... Confidence, D1 ... Line direction, D2 ... Height direction, H0 ... Line area height Now, i ... character spacing candidate, IA1 ... arrangement information, IM1 ... voucher image, NE1 ... network, P1 ... division position, P2 ... half-width division position, P3 ... division position, PR1 ... image processing program, RE9 ... recognition result, U1 ... Extraction unit, U2 ... Processing unit, W1 to W4 ... Width.

Claims (7)

An image processing device that recognizes characters on a voucher image that represents a voucher.
An extraction unit that extracts a plurality of provisional character areas arranged in the line direction from the voucher image,
With a processing unit,
The processing unit
A plurality of character spacing candidates are set as intervals for partitioning the line area including the plurality of provisional character areas in the line direction.
For each of the character spacing candidates, the degree of overlap between the division position according to the character spacing candidate and the character existence portion is calculated.
Based on the calculated multiplicity, the character spacing is determined from the plurality of character spacing candidates.
An image processing device that recognizes characters in a section area in which the line area is divided according to the determined character spacing. The extraction unit associates the arrangement information representing the arrangement of the provisional character area with the provisional character area.
The processing unit determines the height of the line area based on the arrangement information associated with each provisional character area, and sets the plurality of character spacing candidates based on the determined height. Item 1. The image processing apparatus according to item 1. The processing unit sets a half-width character spacing that temporarily divides the line area in half-width character units based on the degree of overlap, and the character existing portion among a plurality of half-width division positions according to the set half-width character spacing. By removing the half-width section position where the degree of overlap is larger than the reference degree from the section of the line area, the half-width character spacing is changed to the full-width character spacing, and the half-width character spacing or the full-width character spacing is one of the above. The image processing apparatus according to claim 1 or 2, wherein characters are recognized for each of the compartmentalized areas according to character spacing. Claims 1 to 3, wherein the processing unit recognizes characters for each of the compartments using a character recognition model generated by machine learning for recognizing characters included in an image representing a voucher. The image processing apparatus according to any one item. The extraction unit extracts the plurality of provisional character areas from the voucher image by performing the first character recognition on the voucher image.
The image processing apparatus according to any one of claims 1 to 4, wherein the processing performs a second character recognition having a higher accuracy than the first character recognition for each of the compartmentalized areas. It is an image processing method that recognizes characters on a voucher image that represents a voucher.
An extraction step of extracting a plurality of provisional character areas arranged in the line direction from the voucher image, and
Including the processing process,
In the processing step,
A plurality of character spacing candidates are set as intervals for partitioning the line area including the plurality of provisional character areas in the line direction.
For each of the character spacing candidates, the degree of overlap between the division position according to the character spacing candidate and the character existence portion is calculated.
Based on the calculated multiplicity, the character spacing is determined from the plurality of character spacing candidates.
An image processing method including a processing step of recognizing characters in a section area in which the line area is divided according to the determined character spacing. An image processing program for character recognition of a voucher image representing a voucher.
An extraction function that extracts a plurality of provisional character areas arranged in the line direction from the voucher image, and
Realize the processing function on the computer,
The processing function
A plurality of character spacing candidates are set as intervals for partitioning the line area including the plurality of provisional character areas in the line direction.
For each of the character spacing candidates, the degree of overlap between the division position according to the character spacing candidate and the character existence portion is calculated.
Based on the calculated multiplicity, the character spacing is determined from the plurality of character spacing candidates.
An image processing program that recognizes characters in a section area that divides the line area according to the determined character spacing.