JP2021149459A - Image processor, control method, and control program - Google Patents

Abstract

To more correctly provide an image processor that can output letter information described in an input ledger sheet image.SOLUTION: The image processor includes: a storage unit 210 for storing a partial image, or a feature amount, positional information, and letter information that were failed to be recognized in the past, for each of types of a plurality of ledger sheet layouts; a layout detection unit 222 for detecting the type of a ledger sheet layout on the basis of an input ledger sheet image; an extraction unit for extracting an input partial image from the input ledger sheet image on the basis of positional information corresponding to the detected type of the ledger sheet layout; and an output unit 203 for determining that the input ledger sheet image describes letter information for the type of the ledger sheet layout and outputting the letter information, when the input partial image and a partial image for the type of the ledger sheet layout are similar or when the feature amount of the input partial image and the feature amount corresponding to the type of the ledger sheet layout are similar.SELECTED DRAWING: Figure 6

Description

The present invention relates to an image processing device, a control method and a control program, and more particularly to an image processing device, a control method and a control program for processing an input form image.

In a company where the person in charge manually converts invoices and other forms into data, the work load of the person in charge becomes heavy when it is necessary to convert a huge number of forms into data, so the efficiency of the form data conversion work is efficient. There is a growing demand for conversion. In order to reduce the work load of the person in charge, it is desired that the image processing device that converts the form into data correctly outputs the character information described in the input form image.

When the pattern such as the orderer's logo or symbol mark detected from the image data is similar to the predetermined pattern, it is determined that the image data is the image data of a specific document such as the FAX image of the purchase order. An information processing device is disclosed (Patent Document 1). When the information processing device determines that the image data is the image data of a specific document, the information processing device specifies the company name associated with the predetermined pattern from the company master DB.

JP-A-2018-42067

It is desired that the image processing apparatus output the character information described in the input form image more accurately.

An object of the present invention is to provide an image processing device, a control method, and a control program capable of more accurately outputting character information described in an input form image.

In the image processing apparatus according to one aspect of the present invention, the partial image that has failed to be recognized in the past or the feature amount of the partial image, the position information of the partial image in the form, and the character information corresponding to the partial image are A storage unit that is stored for each of a plurality of form layout types, an acquisition unit that acquires an input form image, a layout detection unit that detects the type of form layout based on the input form image, and a detected form layout. An extraction unit that extracts an input partial image from an input form image based on the position information stored in the storage unit corresponding to the type of, and a storage unit corresponding to the type of the input partial image and the detected form layout. When the partial image stored in is similar, or when the feature amount of the input partial image is similar to the feature amount of the partial image stored in the storage unit corresponding to the type of the detected form layout. The input form image has an output unit that outputs the character information, assuming that the character information stored in the storage unit is described according to the type of the detected form layout.

The control method according to one aspect of the present invention is a control method of an image processing device having a storage unit and an output unit, and is a partial image or a feature amount of the partial image that the image processing device has failed to recognize in the past. , The position information in the form of the partial image and the character information corresponding to the partial image are stored in the storage unit for each type of a plurality of form layouts, the input form image is acquired, and based on the input form image. , Detects the type of form layout, extracts the input partial image from the input form image based on the position information stored in the storage unit corresponding to the detected form layout type, and detects it as the input partial image. When the partial image stored in the storage unit corresponds to the type of form layout, or when the feature amount of the input partial image and the detected form layout type are stored in the storage unit. When the feature amount of the partial image is similar to that of the partial image, it is assumed that the input partial image contains the character information stored in the storage unit corresponding to the type of the detected form layout. Output from the output section.

The control program according to one aspect of the present invention is a computer control program having a storage unit and an output unit, and is a partial image that has failed to be recognized in the past, a feature amount of the partial image, and a form of the partial image. The position information of the above and the character information corresponding to the partial image are stored in the storage unit for each of a plurality of form layout types, the input form image is acquired, and the type of the form layout is detected based on the input form image. Then, based on the position information stored in the storage unit corresponding to the detected form layout type, the input partial image is extracted from the input form image, and the input partial image and the detected form layout type are supported. When the partial image stored in the storage unit is similar, or the feature amount of the input partial image and the feature amount of the partial image stored in the storage unit corresponding to the detected form layout type. If they are similar, it is assumed that the input partial image contains the character information stored in the storage unit corresponding to the detected form layout type, and the character information is output from the output unit. Let the computer do it.

According to the present invention, the image processing device, the control method, and the control program can more correctly output the character information described in the input form image.

It is a figure which shows the schematic structure of the image processing system 1 according to an embodiment. It is a figure which shows an example of the data structure of a layout table. It is a figure which shows an example of the data structure of a dictionary table. It is a figure which shows the schematic structure of the 2nd storage device 210 and the 2nd processing circuit 220. It is a flowchart which shows the example of the operation of the image reading process. It is a flowchart which shows the example of the operation of the recognition process. It is a flowchart which shows an example of operation of a registration process. It is a schematic diagram which shows an example of the input form image 800. It is a block diagram which shows the schematic structure of another 2nd processing circuit 230.

Hereinafter, the image processing apparatus, the control method, and the control program according to one aspect of the present invention will be described with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to those embodiments, but extends to the inventions described in the claims and their equivalents.

FIG. 1 is a diagram showing a schematic configuration of an image processing system 1 according to an embodiment. As shown in FIG. 1, the image processing system 1 includes an image reading device 100 and an information processing device 200.

The image reading device 100 is, for example, a scanner device or the like. The image reading device 100 is connected to the information processing device 200. The information processing device 200 is an example of an image processing device, such as a personal computer.

The image reading device 100 includes a first interface device 101, an imaging device 102, a first storage device 110, and a first processing circuit 120.

The first interface device 101 has an interface circuit similar to a serial bus such as USB (Universal Serial Bus), and is electrically connected to the information processing device 200 to transmit and receive image data and various kinds of information. Further, instead of the first interface device 101, a communication device having an antenna for transmitting and receiving wireless signals and a wireless communication interface circuit for transmitting and receiving signals through a wireless communication line according to a predetermined communication protocol may be used. good. The predetermined communication protocol is, for example, a wireless LAN (Local Area Network).

The image pickup device 102 has a reduction optical system type image pickup sensor including an image pickup element by CCD (Charge Coupled Device) arranged linearly in the main scanning direction. Further, the image pickup device 102 includes a light source that irradiates light, a lens that forms an image on the image pickup element, and an A / D conversion that amplifies an electric signal output from the image pickup element and performs analog / digital (A / D) conversion. Has a vessel. In the image pickup apparatus 102, the image pickup sensor images the conveyed medium to generate and output an analog image signal, and the A / D converter A / D-converts the analog image signal and digitally inputs it. Generates and outputs a form image. The input form image is a color multi-valued image in which each pixel data is composed of, for example, a total of 24 bits of R (red) value, G (green) value, and B (blue) value represented by 8 bits for each RGB color. Instead of the CCD, a CIS (Contact Image Sensor) of the same magnification optical system type including an image sensor made of CMOS (Complementary Metal Oxide Semiconductor) may be used.

The first storage device 110 includes a memory device such as a RAM (Random Access Memory) and a ROM (Read Only Memory), a fixed disk device such as a hard disk, or a portable storage device such as a flexible disk and an optical disk. Further, the first storage device 110 stores computer programs, databases, tables, etc. used for various processes of the image reading device 100. The computer program may be installed in the first storage device 110 from a computer-readable portable recording medium using a known setup program or the like. The portable recording medium is, for example, a CD-ROM (compact disk read only memory), a DVD-ROM (digital versatile disk read only memory), or the like. In addition, the first storage device 110 stores an input form image or the like generated by the image pickup device 102.

The first processing circuit 120 operates based on a program stored in the first storage device 110 in advance. The first processing circuit 120 is, for example, a CPU (Control Processing Unit). As the first processing circuit 120, a DSP (digital signal processor), an LSI (large scale integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programming Gate Array), or the like may be used.

The first processing circuit 120 is connected to the first interface device 101, the image pickup device 102, the first storage device 110, and the like, and controls each of these parts. The first processing circuit 120 performs media reading control of the image pickup device 102, data transmission / reception control with the information processing device 200 via the first interface device 101, and the like.

The information processing device 200 includes a second interface device 201, an input device 202, a display device 203, a second storage device 210, and a second processing circuit 220. Hereinafter, each part of the information processing apparatus 200 will be described in detail.

The second interface device 201 has an interface circuit similar to that of the first interface device 101 of the image reading device 100, and connects the information processing device 200 and the image reading device 100. Further, instead of the second interface device 201, a communication device having an antenna for transmitting and receiving wireless signals and a wireless communication interface circuit for transmitting and receiving signals through a wireless communication line according to a predetermined communication protocol such as a wireless LAN It may be used.

The input device 202 has an input device such as a keyboard and a mouse, and an interface circuit for acquiring a signal from the input device, and outputs a signal according to the operation of the user to the second processing circuit 220.

The display device 203 is an example of an output unit. The display device 203 includes a display composed of a liquid crystal display, an organic EL (Electro-Luminescence), and the like, and an interface circuit for outputting image data to the display. The display device 203 displays various information on the display according to the instruction from the second processing circuit 220.

The second storage device 210 is an example of a storage unit, and has a memory device, a fixed disk device, a portable storage device, and the like similar to the first storage device 110 of the image reading device 100. The second storage device 210 stores computer programs, databases, tables, and the like used for various processes of the information processing device 200. The computer program may be installed in the second storage device 210 from a computer-readable portable recording medium such as a CD-ROM or a DVD-ROM using a known setup program or the like.

In addition, a layout table, a dictionary table, and the like are stored in advance as data in the second storage device 210. Details of each table will be described later.

The second processing circuit 220 operates based on a program stored in the second storage device 210 in advance. The second processing circuit 220 is, for example, a CPU. A DSP, LSI, ASIC, FPGA, or the like may be used as the second processing circuit 220.

The second processing circuit 220 is connected to the second interface device 201, the input device 202, the display device 203, the second storage device 210, and the like, and controls each of these parts. The second processing circuit 220 performs data transmission / reception control with the image reading device 100 via the second interface device 201, input control of the input device 202, display control of the display device 203, and the like.

FIG. 2 is a diagram showing an example of the data structure of the layout table.

In the layout table, identification information (type ID) of each type, ruled line information corresponding to each type, color information, keyword information, etc. are stored in association with each type of a plurality of form layouts.

The ruled line information indicates the image pattern of the figure indicated by the ruled line included in the form image in which the form is captured and the position of the figure in the form image. For example, the figure indicated by the ruled line is a table, and the image pattern of the entire table is set as the image pattern. As an image pattern, an image pattern at each intersection of a straight line extending in the horizontal direction and a straight line extending in the vertical direction in the table may be set. Further, as the position of the figure, the coordinates of the upper left corner and the lower right corner of the circumscribed rectangle of the figure in the form image are set. As the ruled line information, the position of each straight line extending in the horizontal or vertical direction in the form image may be set. Further, as the ruled line information, image patterns of a plurality of figures and positions in the form image of each figure may be set.

The color information indicates information about the color included in the form image. For example, as color information, for each of the two types of color difference (U, V) in the image, each color difference value (U value, V value) is set as a class, and the number of pixels indicating each color difference value in the form image is set as a frequency. The histogram is set.

The keyword information indicates one or more characters (keywords) and the position of each character in the form image. Characters (keywords) are words such as invoices, receipts, and amounts, especially words used in titles. As the position of the character, the coordinates of the upper left corner and the lower right corner of the circumscribed rectangle of the character in the form image are set.

FIG. 3 is a diagram showing an example of the data structure of the dictionary table.

In the dictionary table, each type ID and one or a plurality of dictionaries corresponding to each type are associated and stored for each type of a plurality of form layouts. The dictionary is an example of an information group, and each includes a partial image, a feature amount, a position information, character information, a priority order, and the like. Each dictionary is set by the user in the registration process described later. A part or all of each dictionary may be set in advance at the time of shipment of the information processing apparatus 200.

The partial image is a part of the input form image in which the recognition of the target character to be detected has failed in the past. As a partial image, an image of an area including features of the input form image, particularly features corresponding to the target characters, is set in the input form image. For example, an image including a target character is set as a partial image. An image that does not include the target character may be set as the partial image.

The feature amount is a feature amount calculated from the corresponding partial image. As the feature amount, for example, A-KAZE feature amount or ORB (Oriented FAST and Rotated Binary Robust Independent Elementary Features) feature amount or the like is used. As the feature amount, other feature amounts such as Haar-Like feature amount and HOG (Histograms of Oriented Gradients) feature amount may be used. The Haar-like feature amount is the difference in luminance value between a plurality of adjacent rectangular areas arbitrarily set in the image area. The HOG feature amount is a histogram of the gradient intensity for each gradient direction of the pixel values of the local region (cell) in the image region. As the feature amount, a plurality of types of feature amount may be set. Moreover, either one of the partial image and the feature amount may be omitted.

The position information indicates the position of the partial image in the form. As the position information, for example, the coordinates of the upper left corner and the lower right corner of the partial image in the input form image are set.

The character information indicates a target character corresponding to the partial image. The target character is a character to be detected in the input form image including the partial image, and is a character (correct answer character) that has failed to be recognized or specified in the past. For example, the character information indicates the actual characters described in the partial image. The character information may indicate actual characters described in an area other than the partial image in the input form image. For example, when the form is an invoice, the company name of the billing source is set as the target character. If the form is a receipt, the issuer's company name or the like is set as the target character.

The priority indicates the order in which each dictionary is referenced.

FIG. 4 is a diagram showing a schematic configuration of the second storage device 210 and the second processing circuit 220.

As shown in FIG. 4, the second storage device 210 includes programs such as an acquisition program 211, a layout detection program 212, an extraction program 213, a character identification program 214, a character recognition program 215, an output control program 216, and a registration program 217. Be remembered. Each of these programs is a functional module implemented by software running on the processor. The second processing circuit 220 reads each program stored in the second storage device 210, and operates according to each read program. As a result, the second processing circuit 220 functions as an acquisition unit 221, a layout detection unit 222, an extraction unit 223, a character identification unit 224, a character recognition unit 225, an output control unit 226, and a registration unit 227.

FIG. 5 is a flowchart showing an example of the operation of the image reading process by the image reading device 100. Hereinafter, the operation of the image reading process will be described with reference to the flowchart shown in FIG. The operation flow described below is mainly executed by the first processing circuit 120 in cooperation with each element of the image reading device 100 based on the program stored in the first storage device 110 in advance.

First, the imaging device 102 captures a form such as an invoice, a notification, or a certificate as a manuscript, generates an input form image, and stores it in the first storage device 110 (step S101).

Next, the first processing circuit 120 transmits the input form image stored in the first storage device 110 to the information processing device 200 via the first interface device 101 (step S102), and ends a series of steps. ..

FIG. 6 is a flowchart showing an example of the operation of the recognition process by the information processing apparatus 200. Hereinafter, the operation of the recognition process will be described with reference to the flowchart shown in FIG. The operation flow described below is mainly executed by the second processing circuit 220 in cooperation with each element of the information processing device 200 based on the program stored in the second storage device 210 in advance.

First, the acquisition unit 221 acquires the input form image from the image reading device 100 via the second interface device 201 and stores it in the second storage device 210 (step S201).

Next, the layout detection unit 222 refers to the layout table and detects the type of the form layout based on the input form image (step S202).

The layout detection unit 222 first detects a ruled line from the input form image. The layout detection unit 222 extracts edge pixels from the input form image and generates an edge image obtained by binarizing the input form image into edge pixels and non-edge pixels. The layout detection unit 222 calculates the absolute value of the difference between the gradation values of the pixels on both sides of the pixel in the horizontal direction in the input form image (hereinafter referred to as the adjacent difference value), and the adjacent difference value exceeds the first threshold value. In this case, the pixels on the input form image are extracted as edge pixels. The gradation value is a luminance value or a color value (R value, G value or B value). The first threshold value can be set, for example, to a difference in brightness value (for example, 20) that allows a person to visually discriminate the difference in brightness on the image. The layout detection unit 222 also calculates the adjacent difference value in the vertical direction, and when the adjacent difference value exceeds the first threshold value, the pixel on the input form image is also extracted as an edge pixel. On the other hand, the layout detection unit 222 extracts the pixels that were not extracted as edge pixels as non-edge pixels.

The layout detection unit 222 may calculate the absolute value of the difference in the gradation values of the pixels horizontally or vertically separated from the pixels in the input form image by a predetermined distance as the adjacent difference value. Further, in the layout detection unit 222, the gradation value of a specific pixel is less than the first threshold value, and the gradation value of a pixel adjacent to the specific pixel or a pixel separated by a predetermined distance from the specific pixel is the first. If it is equal to or more than the threshold value, the specific pixel may be extracted as an edge pixel.

Next, the layout detection unit 222 detects, for example, a region in the edge image in which the non-edge pixels are surrounded by the edge pixels as a cell region corresponding to the cells in the table. The layout detection unit 222 groups the first connected regions surrounded by edge pixels adjacent to each other in the edge image by labeling, and the size of each first connected region in the horizontal or vertical direction is the first size or more. The first connection region is extracted. The first size is set to a size that can include characters inside, and is set to, for example, the number of pixels corresponding to 16 points. The layout detection unit 222 groups the second connection region surrounded by the adjacent non-edge pixels in each extracted first connection region by labeling, and the length of each second connection region in the horizontal or vertical direction is set. The second connecting region having a second size or larger is extracted. The second size is set to the minimum size of characters, for example, the number of pixels corresponding to 8 points. The layout detection unit 222 detects a region (area excluding the second connection region) adjacent to the extracted second connection region and surrounded by edge pixels surrounding the second connection region as a cell region.

The layout detection unit 222 may extract a straight line in which edge pixels are continuous in the edge image, and detect a region surrounded by the extracted straight line (a region excluding the inner region thereof) as a cell region. In that case, the layout detection unit 222 extracts a straight line by using, for example, a morphology transformation. The layout detection unit 222 executes shrinkage processing for converting edge pixels adjacent to non-edge pixels into non-edge pixels in the edge image a predetermined number of times (for the first size) in the horizontal direction, and then performs non-edges adjacent to the edge pixels. The expansion process for converting pixels into edge pixels is executed a predetermined number of times. The layout detection unit 222 extracts the remaining edge pixels as straight lines extending in the horizontal direction. Similarly, the layout detection unit 222 executes a shrinkage process for converting an edge pixel adjacent to a non-edge pixel into a non-edge pixel in the edge image a predetermined number of times, and then performs a shrinkage process to convert the non-edge pixel adjacent to the edge pixel into a non-edge pixel. The expansion process for converting to edge pixels is executed a predetermined number of times. The layout detection unit 222 extracts the remaining edge pixels as straight lines extending in the vertical direction.

Next, the layout detection unit 222 calculates the ruled line similarity between the image pattern shown in the corresponding ruled line information and the cell area detected from the edge image for each type of layout stored in the layout table. The layout detection unit 222 cuts out a region corresponding to the position shown in each ruled line information and having the same size as the image pattern from the edge image while shifting the position, and the cut-out image similar to the image pattern. Calculate the degree. The degree of similarity is, for example, the normalized cross-correlation value. The degree of similarity may be the reciprocal of SSD (Sum of Squared Difference) or the reciprocal of SAD (Sum of Absolute Difference). The layout detection unit 222 calculates the maximum value of the degree of similarity between each cropped image and the image pattern as the ruled line similarity with respect to the layout type. When the image patterns of a plurality of figures are set as the ruled line information, the layout detection unit 222 calculates the average value, the median value, the minimum value, or the maximum value of the similar degree calculated for each image pattern of the plurality of figures. The maximum value is calculated as the ruled line similarity to the form data.

Further, when the position of each straight line extending in the horizontal or vertical direction in the image is set as the ruled line information, the layout detection unit 222 detects the straight line from the input form image by using a known image processing technique. do. The layout detection unit 222 calculates the ratio of the number of straight lines detected from the corresponding positions of the input form image to the total number of straight lines set in the ruled line information for each layout type as the ruled line similarity with respect to the layout type. do.

Further, the layout detection unit 222 generates a histogram in which each color difference value is a class and the number of pixels indicating each color difference value in the input form image is a frequency for each of the two types of color differences in the image. Next, the layout detection unit 222 calculates the color similarity between the histogram shown in the corresponding color information and the histogram generated from the input form image for each type of layout stored in the layout table. The layout detection unit 222 calculates the degree of similarity between each histogram shown in the color information and each histogram generated from the input form image, and calculates the average value or the total value of the calculated degree of similarity for each layout type. It is calculated as the color similarity to. The degree of similarity is, for example, the dot product value of each vector having the frequency of each class of each histogram as an element.

Further, the layout detection unit 222 detects characters from the positions in the input form image corresponding to the positions shown in the corresponding keyword information for each type of layout stored in the layout table. The layout detection unit 222 detects characters by using a known OCR (Optical Character Recognition) technique. The layout detection unit 222 determines whether or not the detected characters match the keywords shown in the keyword information. The layout detection unit 222 calculates the ratio of the number of keywords that match the characters detected from the input form image to the total number of keywords set in the keyword information as the keyword similarity for each layout type.

The layout detection unit 222 calculates the average values of the calculated ruled line similarity, color similarity, and keyword similarity for each layout type stored in the layout table as the similarity for each layout type. In general, different types of layouts are likely to have different arrangements of ruled lines such as tables or straight lines, but are likely to be similar in color. Therefore, the layout detection unit 222 weights the ruled line similarity, the keyword similarity, and the color similarity so that the weight increases in the order of the ruled line similarity, the keyword similarity, and the color similarity as the layout similarity for each layout type. The sum may be calculated. The layout detection unit 222 may calculate the similarity based on any one or two of the ruled line similarity, the color similarity, and the keyword similarity. The layout detection unit 222 detects the type of the form layout having the maximum similarity among the types of the form layout stored in the layout table as the type of the form layout of the input form image.

Next, the extraction unit 223 determines whether or not the dictionary corresponding to the type of form layout detected by the layout detection unit 222 is stored in the dictionary table (step S203). If the dictionary corresponding to the detected form layout type is not stored in the dictionary table, the extraction unit 223 shifts the process to step S209.

On the other hand, when the dictionary corresponding to the detected form layout type is stored in the dictionary table, the extraction unit 223 extracts a specific dictionary from the dictionaries corresponding to the detected form layout type (step). S204). For example, the extraction unit 223 extracts dictionaries in descending order of priority associated with each dictionary.

Next, the extraction unit 223 extracts an input partial image from the input form image based on the position information stored in the extracted dictionary (step S205). The extraction unit 223 extracts an image of a region shown in the position information stored in the extracted dictionary as an input partial image in the input form image. In this way, the extraction unit 223 extracts the input partial image from the input form image based on the position information stored in the dictionary table corresponding to the detected type of form layout.

Next, the character identification unit 224 determines whether or not the extracted input partial image and the partial image stored in the extracted dictionary are similar (step S206). The character identification unit 224 calculates the degree of similarity between the input partial image and the partial image, and determines whether or not the input partial image and the partial image are similar depending on whether or not the calculated degree of similarity is equal to or greater than the threshold value. judge. The degree of similarity is, for example, the normalized cross-correlation value. The degree of similarity may be the reciprocal of SSD or the reciprocal of SAD. In this way, the character identification unit 224 determines whether or not the extracted input partial image and the partial image stored in the dictionary table corresponding to the detected type of form layout are similar.

When the input partial image and the partial image are similar, the character identification unit 224 identifies the character shown in the character information stored in the extracted dictionary as the target character to be detected from the input form image (step S207). The process proceeds to step S210. In this way, when the input partial image and the partial image stored corresponding to the detected form layout type are similar, the character identification unit 224 corresponds to the input form image and the type of the form layout. It is assumed that the character information stored in the above is described.

On the other hand, when the input partial image and the partial image are not similar, the character identification unit 224 determines whether or not there is a dictionary that has not been processed yet among the dictionaries corresponding to the detected form layout type (step). S208).

If there is a dictionary that has not been processed yet, the character identification unit 224 returns the processing to step S204 and repeats the processing of steps S204 to S208. In this case, in step S204, the extraction unit 223 extracts the dictionary having the highest priority among the dictionaries corresponding to the detected form layout type and which have not been processed yet. In step S205, the extraction unit 223 extracts the second input partial image from the input form image based on the position information included in the newly extracted dictionary. When the second input partial image and the partial image included in the newly extracted dictionary are similar, in step S207, the character identification unit 224 is the character indicated in the character information stored in the newly extracted dictionary. Is specified as the target character to be detected from the input form image.

In this way, when the input partial image and the partial image included in the specific dictionary stored in the dictionary table corresponding to the type of the detected form layout are not similar, the extraction unit 223 determines the form layout. Extract other dictionaries stored according to the type. Then, the extraction unit 223 extracts the second input partial image from the input form image based on the position information included in the other extracted dictionaries. When the second input partial image and the partial image included in the other dictionary are similar to each other, the character identification unit 224 assumes that the character information included in the other dictionary is described in the input form image. As a result, the character identification unit 224 can more accurately identify the character to be detected by using a plurality of dictionaries corresponding to different regions.

On the other hand, when all the dictionaries corresponding to the detected form layout types have already been processed, the character recognition unit 225 recognizes the character information indicating the target character from the input form image by OCR (step S209).

For example, the character recognition unit 225 detects a keyword such as "company name" from the input form image by using a known OCR technique. The character recognition unit 225 detects a character having a predetermined positional relationship with respect to the detected character string as a target character. The predetermined positional relationship includes a direction (for example, right side, lower side, lower right side) and a distance (for example, within a pixel corresponding to 30 mm) and is set in advance.

The character recognition unit 225 may detect a character string including a keyword such as "company" from the input form image. The character recognition unit 225 extracts a target character from the detected character strings according to a predetermined priority. The predetermined priority order includes a position condition (for example, a character string located on the rightmost side or the lowest side of the detected character strings) and is set in advance. For example, if the form is an invoice and the target characters are the company name of the billing source, the invoice is likely to include the company name of the billing source and the company name of the billing destination. Generally, the billing company name is likely to be listed on the upper left side, and the billing company name is likely to be listed on the lower right side. Therefore, the character recognition unit 225 can accurately detect the company name of the billing source by detecting the character string located on the rightmost side or the lowermost side of the detected character strings as the target character.

As described above, when the input partial image is not similar to the partial image stored in all the dictionaries corresponding to the detected form layout type, the character recognition unit 225 recognizes the character information from the input form image by OCR. .. Further, the character recognition unit 225 recognizes the character information by OCR from the input form image even when the dictionary corresponding to the detected form layout type is not stored in the dictionary table. As a result, the character recognition unit 225 can recognize the character to be detected even when the dictionary corresponding to the input form image has not been registered yet.

Next, the output control unit 226 outputs the character information specified in step S207 or the character information recognized in step S209 by displaying it on the display device 203 (step S210). The output control unit 226 may output the character information specified in step S207 or the character information recognized in step S209 by transmitting the character information to another information processing device via the second interface device 201. ..

Next, the registration unit 227 receives a designation from the user whether or not the output character information is incorrect using the input device 202 or via the second interface device 201 (step S211).

Next, the registration unit 227 determines that the character information output by the user, that is, the character information stored in the dictionary table specified in step S207 or the character information recognized in step S209 is incorrect. It is determined whether or not (step S212). If the user does not determine that the output character information is incorrect, the registration unit 227 ends a series of steps.

On the other hand, when the user determines that the output character information is incorrect, the registration unit 227 executes the registration process (step S213) and ends a series of steps. That is, when the dictionary corresponding to the detected form layout type is not stored in the dictionary table and the recognized character information is incorrect, the registration unit 227 executes the registration process. If the dictionary corresponding to the detected form layout type is not stored, the registration unit 227 may execute the registration process regardless of whether the recognized character information is correct or incorrect. The details of the registration process will be described later.

In step S206, the character identification unit 224 may determine whether or not the features are similar, instead of determining whether or not the images are similar. In that case, the character identification unit 224 calculates the feature amount of the extracted input partial image, and determines whether or not the calculated feature amount and the feature amount stored in the extracted dictionary are similar. The feature amount calculated by the character identification unit 224 is the same type of feature amount as the feature amount stored in the dictionary. The character identification unit 224 calculates the degree of similarity between the feature amount of the input partial image and the feature amount stored in the dictionary. The character identification unit 224 determines whether or not the feature amount of the input partial image and the feature amount stored in the dictionary are similar depending on whether or not the calculated degree of similarity is equal to or higher than the threshold value. The degree of similarity is, for example, the internal product value of each feature quantity (feature vector).

When the feature amount of the input partial image and the feature amount stored in the dictionary are similar, in step S207, the character identification unit 224 specifies the character shown in the character information stored in the dictionary as the target character. .. If the feature amount of the input partial image and the feature amount stored in the dictionary are not similar, in step S204, the extraction unit 223 newly extracts another dictionary corresponding to the detected form layout type. .. Then, in step S205, the extraction unit 223 extracts the second input partial image from the input form image based on the position information included in the newly extracted dictionary. When the feature amount of the second input partial image and the feature amount stored in the newly extracted dictionary are similar, in step S207, the character identification unit 224 puts the newly extracted dictionary in the input form image. The character shown in the character information stored in is specified as the target character.

In this way, the character identification unit 224 determines whether or not the feature amount of the input partial image is similar to the feature amount of the partial image stored in the dictionary table corresponding to the type of the detected form layout. do. When the feature amount of the input partial image and the feature amount of the partial image stored corresponding to the type of the detected form layout are similar to each other, the character identification unit 224 sets the input form image to the feature amount of the form layout. It is assumed that the character information stored corresponding to the type is described. Further, when the feature amount of the input partial image and the feature amount included in the specific dictionary are not similar, the extraction unit 223 uses another dictionary stored in the dictionary table corresponding to the detected form layout type. Is extracted. Then, the extraction unit 223 extracts the second input partial image from the input form image based on the position information included in the other extracted dictionaries. In the character identification unit 224, when the feature amount of the second input partial image and the feature amount included in the other dictionary are similar, the character information included in the other dictionary is described in the input form image. do.

By using the feature amount, the character identification unit 224 can determine whether or not the input partial image and the partial image are similar with each other more accurately and faster.

Further, in step S207, the character identification unit 224 may specify the target character by using a dictionary in which the partial image most similar to the input partial image is stored. In that case, the character identification unit 224 resembles the input partial image or its feature amount corresponding to each position information and each partial image or its feature amount for all the dictionaries corresponding to the detected form layout type. Calculate the degree. Then, the character identification unit 224 identifies the target character by using a partial image having the maximum degree of similarity or a dictionary in which the feature amount is stored. As a result, the character identification unit 224 can specify the character to be detected with higher accuracy.

Further, in step S202, the layout detection unit 222 may recognize character information from the input form image by using a known OCR technique. In that case, the layout table includes the position in the image in which the character information to be detected is described and the format of the character information (for example, "Co., Ltd." or "Co., Ltd."" in association with each form layout type. Etc.) and are memorized. The layout detection unit 222 recognizes the character information from the position associated with the detected form layout type in the input form image. When the recognized character information corresponds to the format associated with the detected form layout type, the second processing circuit 220 shifts the processing to step S210 and outputs the recognized character information. On the other hand, when the recognized character information does not correspond to the format associated with the detected form layout type, the second processing circuit 220 executes the processes of steps S204 to S209 and specifies the character information using the dictionary. ..

The layout table functions as an overall dictionary in which general information for detecting character information to be detected is stored for each type of form layout. On the other hand, each dictionary stored in the dictionary table functions as an individual dictionary in which detailed information for specifying character information to be detected is stored for each individual local area. The information processing device 200 can accurately identify the character information to be detected by using the general dictionary and the individual dictionary in combination.

FIG. 7 is a flowchart showing an example of the operation of the registration process. The flow of the operation shown in FIG. 7 is executed in step S213 of the flowchart shown in FIG.

First, the registration unit 227 determines whether or not the character information output in step S210 is the character information specified by using the dictionary, that is, whether or not the character information is the character information specified in step S207. (Step S301). If the output character information is not the character information specified by using the dictionary, that is, if the output character information is the character information recognized by OCR, the registration unit 227 shifts the process to step S303.

On the other hand, when the output character information is the character information specified by using the dictionary, the registration unit 227 displays the position indicated by the position information included in the dictionary on the display device 203 in the input form image. Then, the user is notified (step S302). As a result, the user can recognize which area in the input form image the erroneously specified character information was identified based on, and which area in the input form image is newly registered as a dictionary. It is possible to appropriately decide whether to set it in the area. Therefore, the information processing device 200 can improve the quality of the newly registered dictionary.

The information processing device 200 may store the input form image corresponding to each dictionary in the dictionary table, and the registration unit 227 may display the input form image corresponding to the dictionary on the display device 203. As a result, the user can recognize what kind of input form image the erroneously specified character information is based on, and which area in the input form image is newly registered as a dictionary. It is possible to more appropriately decide whether to set to.

Next, the registration unit 227 receives the designation of the area and the target character in the input form image from the user by using the input device 202 (step S303). The area to be designated is an area including a characteristic image in the input form image. The target character to be specified is a character to be detected from the input form image. When the area including the target character is specified by the user, the registration unit 227 can register the partial image in which the actual character information is described as a dictionary, and the character identification unit 224 is the detection target. Characters can be identified more accurately. However, the designated area may be an area that does not include the target character. Further, the designated area may be the same area or an overlapping area as the area already registered in the dictionary.

Next, the registration unit 227 registers the dictionary corresponding to the received area and the target character in the dictionary table (step S304), and ends a series of steps. The registration unit 227 cuts out a partial image corresponding to the received area from the input form image, or calculates the feature amount of the partial image. In addition, the registration unit 227 specifies the position information of the received area in the input form image. In addition, the registration unit 227 generates character information indicating the received target character. Then, the registration unit 227 generates a dictionary including the cut out partial image or the calculated feature amount, the specified position information, and the generated character information. The registration unit 227 sets the priority of the newly generated dictionary to be lower than the priority of all the existing dictionaries. The registration unit 227 may set the priority of the newly generated dictionary to be higher than the priority of all the existing dictionaries. The registration unit 227 registers the generated dictionary in the dictionary table in association with the type of form layout detected by the layout detection unit 222.

The information processing device 200 detects when processing an input form image having the same form format as the input form image by registering a new dictionary based on the input form image that has failed to be recognized or specified. The target character can be specified with high accuracy.

The registration unit 227 may register a plurality of areas for one target character. In that case, in step S303, the registration unit 227 accepts the designation of one target character and a plurality of areas from the user. In step S304, the registration unit 227 generates a dictionary in which a plurality of partial images, a plurality of feature amounts, and a plurality of position information corresponding to the designated plurality of areas are associated with the character information corresponding to the target character. Further, in step S205 of FIG. 6, the extraction unit 223 extracts a plurality of input partial images. In step S206, the character identification unit 224 determines whether or not the plurality of input partial images or their feature amounts are similar to the plurality of partial images or their feature amounts. The character identification unit 224 has a plurality of input partial images depending on whether or not the average value or the weighted sum of each input partial image or its feature amount and the corresponding degree of similarity of each partial image or its feature amount is equal to or more than a threshold value. Alternatively, it is determined whether or not the feature amount and the plurality of partial images or the feature amount are similar.

FIG. 8 is a schematic diagram showing an example of the input form image 800.

As shown in FIG. 8, the input form image 800 is an image in which the invoice is captured. The input form image 800 includes the company name 801 of the billing source and the company name 802 of the billing destination. In the input form image 800, the billing company name 801 is the target character. The seal 803 is superimposed on the company name 801 of the billing source.

For example, if the dictionary corresponding to the type of the form layout of the input form image 800 is not stored in the dictionary table in step S203 of FIG. 6, the character information is recognized in step S209. If two character strings of the billing company name 801 including the keyword "company" and the billing company name 802 are recognized and the billing company name 801 located on the right side is detected as the target character, the step In S212, it is determined that the character information is correct, and the recognition process ends. On the other hand, if the keyword "company" included in the billing company name 801 is not recognized because the seal 803 is superimposed on the billing company name 801, the billing company name 802 is used as the target character. Detected. In that case, the registration process is executed, and for example, the user specifies and specifies the area 804 including the billing company name 801 and the target character "XYZ Development Co., Ltd." indicating the billing company name 801. The dictionary corresponding to the area and the target character is registered in the dictionary table.

On the other hand, when the dictionary corresponding to the type of the form layout of the input form image 800 is stored in the dictionary table in step S203, the dictionary having the highest priority is extracted in step S204. In step S205, the input partial image is extracted from the position indicated by the position information included in the dictionary, and in step S206, it is determined whether or not the input partial image and the partial image included in the dictionary are similar. If the position shown in the position information included in the dictionary indicates the area 804 including the company name 801 of the billing source, the area 804 including the company name 801 of the billing source is extracted as an input partial image. When it is determined that the input partial image and the partial image included in the dictionary are similar, the character information "XYZ Development Co., Ltd." included in the dictionary is output. In that case, it is determined in step S212 that the character information is correct, and the recognition process ends. On the other hand, it was determined that the input partial image and the partial image are not similar because the position of the seal stamp 803 superimposed on the billing company name 801 and the position of the seal stamp in the partial image included in the dictionary are different. If so, the dictionary with the lower priority is extracted in step S204.

If no other dictionary exists, the character information is recognized in step S209. Then, when the recognition of the character information fails, the registration process is executed. In addition, the user does not have the company name 801 of the billing source on which the seal 803 is superimposed, but an area including other features such as the area 805 including the account number of the transfer destination and the company name 801 of the billing source. The target character "XYZ Development Co., Ltd." to be indicated can be specified. In this case as well, the dictionary corresponding to the area specified by the user and the target character is newly registered in the dictionary table.

On the other hand, when another dictionary exists, the second input partial image is extracted from the position indicated by the position information included in the dictionary in step S205, and the second input partial image and the partial image included in the dictionary are included in step S206. It is determined whether or not is similar to. If the position shown in the location information included in the dictionary indicates the area 805 including the transfer destination account, the area 805 including the transfer destination account number is extracted as the second input partial image. When it is determined that the second input partial image and the partial image included in the dictionary are similar, the character information "XYZ Development Co., Ltd." included in the dictionary is output. In that case as well, it is determined in step S212 that the character information is correct, and the recognition process ends.

Further, even if the character information specified by using the registered dictionary is incorrect, the registration process is executed. In that case, in step S302, since the position indicated by the position information included in the dictionary is displayed in the input form image, the user can specify a characteristic area excluding the position. As a result, the information processing apparatus 200 can register a higher quality dictionary.

As described in detail above, the information processing apparatus 200 defines partial images that have failed to be recognized in the past, position information of the partial images, and character information corresponding to the partial images, which correspond to the type of form layout of the input form image. The target character is detected using an individual dictionary. As a result, the information processing apparatus 200 can more correctly output the character information described in the input form image.

As a result, the information processing device 200 can reduce the number of times the input device 202 receives the correction of the character information to be detected from the user, and can reduce the processing load of the processor in the process of handling the input form image. became. Further, when the information processing device 200 transmits information about the character string to be detected to another information processing device, the number of times of receiving a correction request for the character string to be detected from the other information processing device is reduced, and the information processing device 200 receives the correction request. It has become possible to reduce the amount of communication between the 200 and other information processing devices.

In addition, the image processing system 1 can improve the efficiency of the form input work and reduce the work burden of the person in charge in a company that automates the form input work by using OCR and RPA (Robotic Process Automation) technology. It became. In particular, when the image processing system 1 digitizes a form such as an invoice to automate operations such as a payment request, it is necessary to accurately specify the company name of the billing source. However, the billing company name may use a special font with high design rather than a general font, or may have a company seal or the like superimposed on it. In such a case, OCR may not be used to correctly recognize the company name of the billing source. The information processing device 200 can accurately identify the company name of the billing source by using an individual dictionary using partial images that have failed to be recognized in the past. Therefore, the information processing apparatus 200 can appropriately classify and sort the input form images for each billing company name.

In particular, the form layout of invoices, receipts, and other forms differs from company to company, and there is a high possibility that the forms of one company are created according to the same form layout. The information processing device 200 stores a plurality of dictionaries in association with the type of form layout, and changes the dictionary to be used according to the form layout of the input form image. As a result, the information processing apparatus 200 can accurately detect the company name to be detected.

Further, in the information processing apparatus 200, the user can arbitrarily select a partial image to be registered as a dictionary from the input form image that has failed to be recognized. As a result, the user selects an image containing the target character or an image containing information uniquely corresponding to the target character such as a bank account, an address, a telephone number, a fax number, a logo, etc. as a partial image. be able to. Therefore, the information processing apparatus 200 can register a higher quality dictionary.

FIG. 9 is a block diagram showing a schematic configuration of the second processing circuit 230 in the information processing apparatus according to another embodiment.

The second processing circuit 230 executes the recognition process instead of the second processing circuit 220. The second processing circuit 230 includes an acquisition circuit 231, a layout detection circuit 232, an extraction circuit 233, a character identification circuit 234, a character recognition circuit 235, an output control circuit 236, a registration circuit 237, and the like.

The acquisition circuit 231 is an example of the acquisition unit, and has the same function as the acquisition unit 221. The acquisition circuit 231 acquires the input form image from the image reading device 100 via the second interface device 201, and stores it in the second storage device 210.

The layout detection circuit 232 is an example of the layout detection unit, and has the same function as the layout detection unit 222. The layout detection circuit 232 reads the layout table and the input form image from the second storage device 210, detects the type of the form layout, and stores the detection result in the second storage device 210.

The extraction circuit 233 is an example of the extraction unit, and has the same function as the extraction unit 223. The extraction circuit 233 reads the detection result of the dictionary table, the input form image, and the type of the form layout from the second storage device 210, extracts the input partial image from the input form image, and stores it in the second storage device 210.

The character identification circuit 234 is an example of the character identification unit, and has the same function as the character identification unit 224. The character identification circuit 234 reads a dictionary table, an input form image, and an input partial image from the second storage device 210, specifies character information according to whether or not the input partial image and the partial image are similar, and obtains a specific result. 2 Store in the storage device 210.

The character recognition circuit 235 is an example of the character recognition unit, and has the same function as the character recognition unit 225. The character recognition circuit 235 reads the input form image from the second storage device 210, recognizes the character information, and stores the recognition result in the second storage device 210.

The output control circuit 236 is an example of the output control unit, and has the same function as the output control unit 226. The output control circuit 236 reads the character information specific result and the recognition result from the second storage device 210, and outputs the character information to the display device 203.

The registration circuit 237 is an example of the registration unit and has the same function as the registration unit 227. The registration circuit 237 receives the designation of the area and the target character in the input form image from the input device 202, and registers the dictionary corresponding to each received information in the second storage device 210.

As described in detail above, the information processing apparatus can more accurately output the character information described in the input form image even when the second processing circuit 230 is used.

Although suitable embodiments have been described above, the embodiments are not limited thereto. For example, the division of functions between the image reading device 100 and the information processing device 200 is not limited to the example of the image processing system 1 shown in FIG. 1, and each part of the image reading device 100 and the information processing device 200 is divided into the image reading device 100 and the information processing device. Which of the 200 is arranged can be changed as appropriate. Alternatively, the image reading device 100 and the information processing device 200 may be configured by one device.

For example, the first storage device 110 of the image reading device 100 may store each program and each data stored in the second storage device 210 of the information processing device 200. Further, the first processing circuit 120 of the image reading device 100 may operate as each part realized by the second processing circuit 220 of the information processing device 200. Further, the image reading device 100 may have a processing circuit similar to that of the second processing circuit 230 of the information processing device 200.

In that case, the image reading device 100 has an input device similar to the input device 202 and a display device similar to the display device 203. Since the recognition process is executed by the image reading device 100, the process of transmitting and receiving the form image in steps S102 and S201 is omitted. Each process of steps S202 to S213 is executed by the first processing circuit 120 of the image reading device 100. The operation of these processes is the same as that executed by the second processing circuit 220 or the second processing circuit 230 of the information processing apparatus 200. In this case, the image reading device 100 operates as an image processing device.

Further, in the image processing system 1, the first interface device 101 and the second interface device 201 are connected via a network such as the Internet, a telephone line network (including a mobile terminal line network and a general telephone line network), and an intranet. May be good. In that case, the first interface device 101 and the second interface device 201 are provided with a communication interface circuit of the network to be connected. In that case, a plurality of information processing devices are distributed and arranged on the network so that the image processing service can be provided in the form of cloud computing, and each information processing device cooperates to perform recognition processing and the like. You may share it. As a result, the image processing system 1 can efficiently execute the recognition process for the form image read by the plurality of image reading devices.

200 Information processing device 203 Display device 210 Second storage device 221 Acquisition unit 222 Layout detection unit 223 Extraction unit 225 Character recognition unit 227 Registration unit

Claims (8)

The partial image or the feature amount of the partial image that failed to be recognized in the past, the position information in the form of the partial image, and the character information corresponding to the partial image are stored for each type of a plurality of form layouts. Memory and
The acquisition unit that acquires the input form image and
A layout detection unit that detects the type of form layout based on the input form image, and
An extraction unit that extracts an input partial image from the input form image based on the position information stored in the storage unit according to the type of the detected form layout.
When the input partial image and the partial image stored in the storage unit corresponding to the type of the detected form layout are similar, or when the feature amount of the input partial image and the detected form layout are used. When the feature amount of the partial image stored in the storage unit is similar to the type of the input form image, the input form image is stored in the storage unit according to the type of the detected form layout. Assuming that the character information is described, an output unit that outputs the character information and
An image processing device characterized by having. The image processing apparatus according to claim 1, wherein the character information corresponding to the partial image indicates an actual character described in the partial image. When the partial image or feature amount, position information, and character information corresponding to the detected type of form layout are not stored in the storage unit, the area and the target character are designated in the input form image, and the area is concerned. A registration unit that registers a partial image or a feature amount of the partial image, position information of the area, and character information indicating the target character in the storage unit in association with the type of the detected form layout. The image processing apparatus according to claim 1 or 2, further comprising. When the partial image or feature amount, position information, and character information corresponding to the detected form layout type are not stored in the storage unit, a character recognition unit that recognizes character information by OCR from the input form image is further added. Have and
The output unit outputs the recognized character information and outputs the recognized character information.
The image processing device according to claim 3, wherein the registration unit accepts designation of an area and a target character in the input form image when it is determined by the user that the recognized character information is incorrect. When the user determines that the character information stored in the storage unit output from the output unit is incorrect, the registration unit accepts designation of an area and a target character in the input form image. , The partial image corresponding to the area or the feature amount of the partial image, the position information of the area, and the character information indicating the target character are further associated with the type of the detected form layout and registered in the storage unit. The image processing apparatus according to claim 3 or 4. The storage unit contains a plurality of information groups including a partial image that has failed to be recognized in the past, a feature amount of the partial image, position information of the partial image in a form, and character information corresponding to the partial image. , Stored for each type of the plurality of form layouts
When the input partial image and the partial image included in the specific information group stored in the storage unit corresponding to the type of the detected form layout are not similar, or the feature amount of the input partial image. And the feature amount contained in the specific information group are not similar to each other.
The extraction unit obtains a second input partial image from the input form image based on the position information included in other information groups stored in the storage unit according to the type of the detected form layout. Extract and
When the second input partial image and the partial image included in the other information group are similar to each other, or the feature amount of the second input partial image and the partial image included in the other information group are included in the output unit. When the feature amount is similar, it is assumed that the character information included in the other information group is described in the input form image, and the character information is output. Any one of claims 1 to 5. The image processing apparatus according to the section. A control method for an image processing device having a storage unit and an output unit, wherein the image processing device
The partial image or the feature amount of the partial image that failed to be recognized in the past, the position information in the form of the partial image, and the character information corresponding to the partial image are stored in the storage unit for each type of a plurality of form layouts. Remember,
Get the input form image and
Based on the input form image, the type of form layout is detected.
An input partial image is extracted from the input form image based on the position information stored in the storage unit corresponding to the type of the detected form layout.
When the input partial image and the partial image stored in the storage unit corresponding to the type of the detected form layout are similar, or when the feature amount of the input partial image and the detected form layout are used. When the feature amount of the partial image stored in the storage unit is similar to the type of the input partial image, the input partial image is stored in the storage unit according to the type of the detected form layout. Assuming that the character information is described, the character information is output from the output unit.
A control method characterized by that. A computer control program having a storage unit and an output unit.
The partial image or the feature amount of the partial image that failed to be recognized in the past, the position information in the form of the partial image, and the character information corresponding to the partial image are stored in the storage unit for each type of a plurality of form layouts. Remember,
Get the input form image and
Based on the input form image, the type of form layout is detected.
An input partial image is extracted from the input form image based on the position information stored in the storage unit corresponding to the type of the detected form layout.
When the input partial image and the partial image stored in the storage unit corresponding to the type of the detected form layout are similar, or when the feature amount of the input partial image and the detected form layout are used. When the feature amount of the partial image stored in the storage unit is similar to the type of the input partial image, the input partial image is stored in the storage unit according to the type of the detected form layout. Assuming that the character information is described, the character information is output from the output unit.
A control program characterized by causing the computer to execute such a thing.

Images