JP7039693B2 - Inspection equipment, control method and control program - Google Patents

Description

The present disclosure relates to an inspection device, a control method and a control program, and more particularly to an inspection device, a control method and a control program for inspecting a form image.

At financial institutions, life insurance companies, non-life insurance companies, etc., the person in charge manually converts the forms such as receipts provided by customers into data. Depending on each institution or company, a huge number of forms are provided by customers, so the work burden on the person in charge is heavy, and there is an increasing demand for more efficient form data conversion work. Therefore, in recent years, OCR (Optical Character Recognition) technology for reading characters from a form has been used. Furthermore, recently, RPA (Robotic Process Automation) technology that automates the work of the person in charge using the characters read from the form has also attracted attention.

A digital copier that recognizes a character string contained in a tablespace extracted from an image is disclosed. This digital copier calculates numerical data obtained by converting a character string consisting of numbers based on an arithmetic expression defined based on a specific character string, and compares the calculated value with the numerical data to be checked. Then, when the calculated value and the numerical data to be checked do not match as a result of the check, the digital copying machine notifies the user to that effect (see Patent Document 1).

Japanese Unexamined Patent Publication No. 2001-325562

Generally, a form such as a receipt contains a plurality of items, and each form has a wide variety of formats. In an apparatus for processing a form image in which a form having such a wide variety of formats is captured, it is desired to be able to more accurately determine whether or not the item value of each item is correctly detected.

The purpose of the inspection device, the control method, and the control program is to make it possible to more accurately determine whether or not the item value is correctly detected from the form image.

The inspection device according to one aspect of the embodiment includes a storage unit that stores item name candidates for each of a plurality of element items in order of priority, and stores relationship information indicating a relationship between item values of a plurality of specific items. The image acquisition unit that acquires a form image that includes any of the item name candidates, and for each of a plurality of element items, it is sequentially determined in order of priority whether the corresponding item name candidate is included in the form image, and is included first. The relationship in which the item values of the detection unit that detects the item name candidate determined to be the same as the specific item corresponding to the element item and the item values of the plurality of specific items detected by the detection unit are shown in the relationship information. It has a determination unit for determining whether or not the determination unit has, and a display control unit for displaying the determination result by the determination unit.

Further, the control method according to one aspect of the embodiment is a control method of an inspection device having a storage unit, in which item name candidates are stored in the storage unit in order of priority for each of a plurality of element items, and a plurality of specific items are stored. The relationship information indicating the relationship between the item values of is stored in the storage unit, the form image including any of the item name candidates is acquired, and the corresponding item name candidate is included in the form image for each of a plurality of element items. It is sequentially determined according to the priority, and the item name candidate determined to be included first is detected as a specific item corresponding to the element item, and each item of the plurality of specific items detected by the detection unit. It includes determining whether or not the value has the relationship shown in the relationship information, and displaying the determination result by the determination unit.

Further, the control program according to one aspect of the embodiment is a control program of an inspection device having a storage unit, and item name candidates are stored in the storage unit in order of priority for each of a plurality of element items, and a plurality of specific items are stored. The relationship information indicating the relationship between the item values of is stored in the storage unit, the form image including any of the item name candidates is acquired, and the corresponding item name candidate is included in the form image for each of a plurality of element items. It is sequentially determined according to the priority, and the item name candidate determined to be included first is detected as a specific item corresponding to the element item, and each item of the plurality of specific items detected by the detection unit. It is determined whether or not the value has the relationship shown in the relationship information, and the inspection device is made to display the determination result by the determination unit.

According to the present embodiment, the inspection device, the control method, and the control program can more accurately determine whether or not the item value is correctly detected from the form image.

The objects and effects of the present invention will be recognized and obtained specifically by using the components and combinations pointed out in the claims. Both the general description described above and the detailed description described below are exemplary and descriptive and do not limit the invention described in the claims.

It is a figure which shows an example of the schematic structure of the inspection apparatus 100. It is a figure which shows the schematic structure of the storage device 110 and the CPU 120. It is a figure which shows an example of the form image 300. It is a figure which shows an example of the form image 400. It is a figure which shows an example of the data structure of a priority table 500. It is a figure which shows an example of the data structure of the arrangement table 510. It is a figure which shows an example of the data structure of the relation table 520. It is a figure which shows an example of the data structure of the detection result table 600. It is a figure which shows an example of the data structure of the determination result table 610. It is a flowchart which shows the example of the operation of a determination process. It is a flowchart which shows the example of the operation of a display process. It is a schematic diagram which shows an example of display data 900. It is a flowchart which shows the example of the operation of a reception process. It is a schematic diagram which shows an example of display data 1100. It is a schematic diagram which shows an example of another display data 1200. It is a block diagram which shows the schematic structure of the processing circuit 130.

Hereinafter, the inspection device according to one aspect of the present disclosure will be described with reference to the drawings. However, it should be noted that the technical scope of the present disclosure is not limited to those embodiments, but extends to the inventions described in the claims and their equivalents.

FIG. 1 is a diagram showing an example of a schematic configuration of an inspection device 100 according to an embodiment.

The inspection device 100 is a portable device such as a tablet PC or a multifunctional mobile phone (so-called smartphone), and is used by the user to process a form image. The form image is image data representing a form. In this embodiment, a form having various layouts is used as the form. A form with various layouts is a form that is individually created by a plurality of different institutions, companies, etc., has a different description position for each item, and has a plurality of synonyms for each item.

The inspection device 100 includes a communication device 101, an input device 102, a display device 103, an image pickup device 104, a storage device 110, a CPU (Central Processing Unit) 120, and a processing circuit 130. Hereinafter, each part of the inspection device 100 will be described in detail.

The communication device 101 has a communication interface circuit including an antenna whose sensitive band is mainly a 2.4 GHz band, a 5 GHz band, or the like. The communication device 101 performs wireless communication with an access point or the like based on a wireless communication method of the IEEE (The Institute of Electrical and Electronics Engineers, Inc.) 802.11 standard. Then, the communication device 101 transmits / receives data to / from an external device (not shown) via the access point. The communication device 101 supplies the data received from the external device via the access point to the CPU 120, and transmits the data supplied from the CPU 120 to the external device via the access point. The communication device 101 may be any device as long as it can communicate with the external device. For example, the communication device 101 may communicate with an external device via a base station device (not shown) according to a mobile phone communication method, or may communicate with an external device according to a wired LAN communication method.

The input device 102 is an example of an operation unit, and has a touch panel type input device, an input device such as a keyboard and a mouse, and an interface circuit for acquiring a signal from the input device. The input device 102 receives the input data input by the user and outputs a signal corresponding to the received input data to the CPU 120.

The display device 103 is an example of a display unit, and has a display composed of a liquid crystal display, an organic EL (Electro-Luminescence), and an interface circuit for outputting image data or various information to the display. The display device 103 is connected to the CPU 120 and displays the image data output from the CPU 120 on the display. The input device 102 and the display device 103 may be integrally configured by using the touch panel display.

The image pickup device 104 includes a reduction optical system type image pickup sensor including an image pickup element composed of CCDs (Charge Coupled Devices) arranged one-dimensionally or two-dimensionally, and an A / D converter. The image pickup apparatus 104 is an example of an image pickup unit, and sequentially captures a form according to an instruction from the CPU 120 and sequentially generates a form image (for example, 30 frames / sec). The image pickup sensor generates an captured analog image signal and outputs it to an A / D converter. The A / D converter converts the output analog image signal into analog-digital, sequentially generates digital image data, and outputs the digital image data to the CPU 120. Instead of the CCD, a 1x optical system type CIS (Contact Image Sensor) provided with an image sensor made of CMOS (Complementary Metal Oxide Semiconductor) may be used. Hereinafter, the digital image data captured and output by the image pickup apparatus 104 will be referred to as a form image.

The storage device 110 is an example of a storage unit. The storage device 110 includes a memory device such as a RAM (Random Access Memory) and a ROM (Read Only Memory), a fixed disk device such as a hard disk, or a portable storage device such as a flexible disk and an optical disk. Further, the storage device 110 stores computer programs, databases, tables, etc. used for various processes of the inspection device 100. The computer program may be installed from a computer-readable portable recording medium such as a CD-ROM (compact disk ROM) or a DVD-ROM (digital versatile disk ROM). The computer program is installed in the storage device 110 using a known setup program or the like. Further, the storage device 110 stores, as data, a priority table, an arrangement table, a relation table, a detection result table, a determination result table, and the like. Details of the priority table, the arrangement table, the relation table, the detection result table, and the judgment result table will be described later.

The CPU 120 operates based on a program stored in the storage device 110 in advance. The CPU 120 may be a general-purpose processor. Instead of the CPU 120, a DSP (digital signal processor), an LSI (large scale integration), or the like may be used. Further, instead of the CPU 120, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or the like may be used.

The CPU 120 is connected to a communication device 101, an input device 102, a display device 103, an image pickup device 104, and a storage device 110, and controls each of these parts. The CPU 120 performs data transmission / reception control via the communication device 101, input control of the input device 102, output control of the display device 103, image pickup control of the image pickup device 104, control of the storage device 110, and the like. Further, the CPU 120 processes the form image.

FIG. 2 is a diagram showing a schematic configuration of a storage device 110 and a CPU 120 of the inspection device 100.

As shown in FIG. 2, each program such as an image acquisition program 111, a detection program 112, a determination program 113, and a display control program 114 is stored in the storage device 110. Each of these programs is a functional module implemented by software running on the processor. The CPU 120 reads each program stored in the storage device 110 and operates according to each read program. As a result, the CPU 120 functions as an image acquisition unit 121, a detection unit 122, a determination unit 123, and a display control unit 124.

FIG. 3 is a diagram showing an example of a form image 300 processed by the inspection device 100, and FIG. 4 is a diagram showing an example of a form image 400 processed by the inspection device 100.

Invoices issued by a predetermined institution, company, or the like are imaged as forms in the form images 300 and 400 shown in FIGS. 3 and 4. In each form, the item value 302 corresponding to each element item 301 is represented for each of the plurality of element items 301. The element item 301 is an item in which each information represented by the form is classified according to a predetermined rule.

The form images 300 and 400 include a quantity, a unit price, an amount and a billing amount (tax included) as element items 301. The item names of each element item of quantity, unit price, amount, and billing amount (tax included) are "unit number", "unit price", "charge", and "total" in the form image 300, and "quantity" in the form image 400. , "Unit price", "Amount", "Total". Further, each element item 301 of quantity, unit price, and amount is arranged in the order of unit price, quantity, and amount from the left in the form image 300, and is arranged in the order of quantity, unit price, and amount from the left in the form image 400. As described above, in the present embodiment, the forms are individually created by a plurality of different institutions, companies, etc., the description positions of the element items 301 are different for each form, and there are a plurality of synonyms for each element item 301. The layout form is used.

In the form images 300 and 400, only one item value 302 is shown in the element item 301 of the billing amount (tax included). On the other hand, in each element item 301 of quantity, unit price, and amount, an item value 302 is shown for each of the plurality of expense items 303, and a plurality of item values 302 are shown for each element item 301. The form images 300 and 400 include each product and consumption tax as the expense item 303.

FIG. 5A is a diagram showing an example of the data structure of the priority table 500.

The priority table 500 is preset by the provider of the inspection device 100 and stored in the storage device 110. As shown in FIG. 5A, in the priority table 500, the identification information (item ID) of each element item, the item content, and one or more item name candidates are stored in association with each other for each of the plurality of element items. To. Each item name candidate is stored in order of priority.

The item ID is a number for identifying each element item. The item content indicates the content of each element item. The item name candidate is a name that is inspected whether or not it is included in the form image when the item value of the element item is detected from the form image, and the name of each element item (item name) described in the form image. ) Candidate. The priority order is an order for defining the order in which the item name candidates specified in the priority table 500 are inspected when the item values of the element items are detected from the form image. In this example, the priority of the numerical value "1" is the highest, and the higher the numerical value, the lower the priority, so that the priority is assigned to each item name candidate.

FIG. 5B is a diagram showing an example of the data structure of the arrangement table 510.

The arrangement table 510 is preset by the provider of the inspection device 100 and stored in the storage device 110. As shown in FIG. 5B, in the arrangement table 510, the item ID of each element item, the arrangement information, and the like are stored in association with each of the plurality of element items.

The arrangement information is information for defining the display positional relationship between the item name of each specific item and the item value corresponding to each item name in the form image. The specific item is an item name candidate actually detected from the form image among the item name candidates of each element item defined in the priority table 500 for each element item. As the arrangement information, for example, the relative position of the item value with respect to the item name (for example, a predetermined range in the upper right, the lower right, the right side, the left side, the upper right side, the lower right side, the upper left side, and the lower left side) and the condition (for example, a numerical string). It is stipulated. The arrangement information is used to specify the item value corresponding to the specific item when detecting the item value of the specific item from the form image.

FIG. 5C is a diagram showing an example of the data structure of the relation table 520.

The relationship table 520 is preset by the provider of the inspection device 100 and stored in the storage device 110. As shown in FIG. 5C, the relationship table 520 stores identification information (relationship ID), relationship information, and the like of each relationship for each relationship between item values of a plurality of specific items.

The relationship ID is a number for identifying each relationship. The relationship information indicates the relationship between the item values of a plurality of specific items. For example, the relationship information shows the relationship between the item value of one specific item and the item value of two or more other specific items. In the example shown in FIG. 5C, for a combination of three item values (amount, quantity, unit price) having the same relative position with respect to the item name of each specific item, the item value (amount) of one specific item is the other two. It is shown to be equal to the product of the item values (quantity, unit price) of one specific item. The relationship information may indicate the relationship between the item value of one specific item and the plurality of item values of another specific item. In the example shown in FIG. 5C, it is shown that the item value (billing amount (tax included)) of one specific item is equal to the sum of the item values (amount of each expense item) of the other specific item. Further, the relationship information may show the relationship between the item value of one specific item and the item value of another specific item. In that case, for example, the item value (billing amount (tax included)) of one specific item is equal to the product of the item value (billing amount (tax excluded)) of the other one specific item and the predetermined value (1 + consumption tax rate). Is shown.

FIG. 6A is a diagram showing an example of the data structure of the detection result table 600.

The detection result table 600 is created for each form image processed by the inspection device 100 in the determination process and stored in the storage device 110. As shown in FIG. 6A, in the detection result table 600, the item ID, the item name of the specific item, the item value, the relative position of the item value, and the like are stored in association with each element item stored in the priority table. ..

The item name of the specific item is the item name of the specific item detected for each element item in the determination process. The item value is an item value detected for each specific item in the determination process. As the relative position of the item value, the horizontal position and the vertical position (coordinates) of the center of gravity position of each item value as seen from the center of gravity position of the item name of each specific item in the form image are stored. When a plurality of item values are detected for each specific item, the item value and the relative position are stored for each detected item value.

FIG. 6B is a diagram showing an example of the data structure of the determination result table 610.

The determination result table 610 is created for each form image processed by the inspection device 100 in the determination process and stored in the storage device 110. As shown in FIG. 6B, in the determination result table 610, each relation ID stored in the relation table 520, the identification information of each inspection target (inspection target ID), the determination result, and the like are stored in association with each other.

The inspection target ID is a number for identifying each inspection target. The inspection target is a target to be inspected whether or not the item value of each specific item has the relationship shown in the relational information. A combination of item values of each specific item is set as an inspection target. The determination result indicates whether or not each detected item value has each relationship for each inspection target. If they have each relationship, normal is set as a judgment result, and if they do not have each relationship, an abnormality is set as a judgment result.

FIG. 7 is a flowchart showing an example of the operation of the determination process by the inspection device 100.

Hereinafter, an example of the operation of the determination process by the inspection device 100 will be described with reference to the flowchart shown in FIG. The operation flow described below is mainly executed by the CPU 120 in cooperation with each element of the inspection device 100 based on the program stored in the storage device 110 in advance.

First, the image acquisition unit 121 causes the image pickup device 104 to image the form to generate the form image, acquires the form image from the image pickup device 104, and stores the acquired form image in the storage device 110 (step S101). The form to be imaged by the image pickup apparatus 104 is a form having various layouts as shown in FIGS. 3 and 4, and the form image includes any of the item name candidates of each element item.

Next, the detection unit 122 detects each character string included in the form image (step S102). The detection unit 122 detects characters from the form image by using a known OCR technique, and groups characters whose detected positions are within a predetermined distance and detects them as a character string. The detection unit 122 stores each detected character string in the storage device 110 in association with the arrangement position in the form image.

The detection unit 122 executes the processes of steps S103 to S109 for each element item stored in the priority table 500. First, the detection unit 122 reads out the priority table 500 and extracts any element item from the priority table 500 (step S103).

The detection unit 122 executes the processes of steps S104 to S106 for each item name candidate associated with the extracted element item. First, the detection unit 122 extracts the item name candidates of the extracted element items from the priority table 500 according to the priority (step S104). The detection unit 122 extracts the item name candidates stored in association with the extracted element items one by one in descending order of priority.

Next, the detection unit 122 determines whether or not the extracted item name candidate is included in any of the character strings detected from the form image (step S105).

If the extracted item name candidate is not included in any of the character strings detected from the form image, the detection unit 122 has already extracted all the item name candidates of the extracted element items from the priority table 500. (Step S106). When all the item name candidates have been extracted, the detection unit 122 determines that the determination process has failed, displays that fact on the display device 103, and ends a series of steps. On the other hand, when all the item name candidates have not been extracted yet, the detection unit 122 returns the process to step S104 and repeats the processes of steps S104 to S105.

On the other hand, when the extracted item name candidate is included in any of the character strings detected from the form image, the detection unit 122 detects the item name candidate as a specific item corresponding to the element item (step S107). ). The detection unit 122 stores the detected item name candidate in the detection result table 600 as a specific item.

Next, the detection unit 122 detects the item value corresponding to the detected specific item (step S108). The detection unit 122 reads out the arrangement information of the element item in which the specific item is detected from the arrangement table 510. The detection unit 122 sets a character string that exists at a relative position shown in the placement information and satisfies the condition shown in the placement information with respect to the position in the form image in which the specific item is detected as the detected specific item. Detect as the corresponding item value. The detection unit 122 stores the detected item value and the relative position of the item value with respect to the position in the form image in which the specific item is detected in the detection result table 600 in association with each specific item. The detection unit 122 detects the ruled lines of the table from the form image by using a known image processing technique, and identifies and specifies the range including the item values corresponding to the specific items based on the detected ruled lines. The item value may be detected only within the specified range.

When a plurality of inspection targets are included in the form image and the detection unit 122 detects an item value for each of the plurality of inspection targets, the detection result table associates each item value and the relative position of each item value with each specific item. Store in 600. That is, the detection unit 122 detects the item value of the specific item corresponding to each element item for each of the plurality of inspection targets.

Next, the detection unit 122 determines whether or not the processing has been executed for all the element items stored in the priority table 500 (step S109). If the processing has not been executed for all the element items, the detection unit 122 returns the processing to step S103 and repeats the processing of steps S103 to S109. In this way, the detection unit 122 sequentially determines whether or not the corresponding item name candidate is included in the form image for each of the plurality of element items in order of priority, and the item name candidate determined to be included first is determined. Is detected as a specific item corresponding to the element item.

On the other hand, when the detection unit 122 executes the process for all the element items, the determination unit 123 extracts the relation information from the relation table 520. Then, the determination unit 123 extracts from the detection result table 600 a combination of item values of each specific item to be inspected for inspection based on the extracted relationship information (step S110). When there are a plurality of combinations of item values to be inspected, the determination unit 123 extracts each combination one by one.

Next, the determination unit 123 determines whether or not the item value of each extracted specific item has the relationship shown in the extracted relationship information (step S111).

When the item value of each specific item has the relationship shown in the relationship information, the determination unit 123 determines that the character recognition process executed in step S102 is correct (step S112). On the other hand, when the item values of each specific item do not have the relationship shown in the relationship information, the determination unit 123 determines that the character recognition process executed in step S102 is incorrect (step S113).

Next, the determination unit 123 generates an inspection target ID, and stores the determination result in the determination result table 610 in association with the relation ID corresponding to the relation information and the generated inspection target ID (step S114). When the determination unit 123 determines that the character recognition process is correct, it stores normal as a determination result, and when it determines that the character recognition process is incorrect, it stores an abnormality as a determination result.

Next, the determination unit 123 determines whether or not the inspection for all the inspection targets is completed based on all the relational information stored in the relational table 520 (step S115). If the inspection for all the inspection targets has not been completed yet, the determination unit 123 returns the process to step S110 and repeats the processes of steps S110 to S114. In this way, the determination unit 123 determines whether or not the item value of each specific item detected by the detection unit 122 has the relationship shown in the relationship information. On the other hand, when the inspection for all the inspection targets is completed, the determination unit 123 ends a series of steps.

The determination unit 123 determines whether or not there is an error in the character recognition processing for each form image, and the form image determined that there is no error in the character recognition processing is transmitted to the RPA processing device (not shown) via the communication device 101. ) May be sent. As a result, the RPA processing device can automatically acquire a highly reliable form image and execute a predetermined RPA processing.

When the determination process is executed for the form image 300 shown in FIG. 3, in step S107, "number of units", "unit price", "charge", and "total" are detected as specific items. In step S108, the item values corresponding to the products A to J are detected for the "number of units", the "unit price", and the "charge", and one item value (64,800) is detected for the "total". Then, in step S111, it is determined whether the "number of units x unit price = charge" is established for the products A to J and whether the "total charge = total" is satisfied.

On the other hand, when the determination process is executed for the form image 400 shown in FIG. 4, "quantity", "unit price", "amount", and "total" are detected as specific items in step S107. In step S108, the item values corresponding to the products A to J are detected for the "quantity" and the "unit price", and the item values corresponding to the products A to J and the consumption tax are detected for the "amount", and the "total" is detected. For, one item value (64,800) is detected. Then, in step S111, it is determined whether "quantity x unit price = amount of money" is satisfied and "total amount of money = total amount" is satisfied for the products A to J.

In various layout forms, terms that have similar meanings such as "total" and "total" have different meanings such as "billing amount (tax included)" and "billing amount (tax excluded)". It may be used as an item name to indicate. For example, if the item name of the element item "billing amount (tax included)" is fixed to "total" and the item value is detected, the corresponding item value is correctly detected from the form image 300, but the corresponding item value is detected from the form image 400. The item value to be used is not detected correctly. On the other hand, if the item name of the element item "billing amount (tax included)" is fixed to "total" and the item value is detected, the corresponding item value is correctly detected from the form image 400, but it corresponds from the form image 300. The item value to be used is not detected correctly.

On the other hand, the inspection device 100 sequentially determines whether the item name candidates of each element item are included in the form image in order of priority, and detects the item name candidate determined to be included first as a specific item. As a result, the inspection device 100 accurately determines whether or not the item values are correctly detected from the form images for each of the form images 300 and 400 that image the forms of various layouts in which the names of the items are different for each form. It can be determined.

FIG. 8 is a flowchart showing an example of the operation of the display process by the inspection device 100.

Hereinafter, an example of the operation of the display process by the inspection device 100 will be described with reference to the flowchart shown in FIG. The operation flow described below is mainly executed by the CPU 120 in cooperation with each element of the inspection device 100 based on the program stored in the storage device 110 in advance. The display process is executed when a request for displaying the determination result using the input device 102 by the user is received.

First, the display control unit 124 extracts the detection result table 600 and the determination result table 610 corresponding to all the form images for which the determination process has been executed from the storage device 110 (step S201).

Next, the display control unit 124 reads the determination result from each of the extracted determination result tables 610, and identifies the form image having an error in the character recognition process as the recognition error image (step S202).

Next, the display control unit 124 determines one specific image as the image of interest from the identified recognition error images (step S203). For example, the display control unit 124 determines the image of interest that has the earliest (previous) time when the determination process is executed among the recognition error images. If there is no form image with an error in the character recognition process, the display control unit 124 sets a specific image as the image of interest from the form images with no error in the character recognition process.

Next, the display control unit 124 reads the determination result from the determination result table 610 corresponding to the image of interest, and identifies the inspection target for which the character recognition process is incorrect as the recognition error target (step S204).

Next, the display control unit 124 determines one specific inspection target as the attention target from the identified recognition error targets (step S205). For example, the display control unit 124 determines the inspection target stored at the top of the determination result table 610 as the target of interest among the recognition error targets. When there is no inspection target having an error in the character recognition process, the display control unit 124 sets one specific inspection target as the attention target from the inspection targets having no error in the character recognition process.

Next, the display control unit 124 displays a display for displaying the determination result by the determination unit 123 based on the extracted form image, the specified recognition error image and the recognition error target, and the determined attention image and attention target. Create data. Then, the display control unit 124 displays the created display data on the display device 103 (step S206).

FIG. 9 is a schematic diagram showing an example of display data 900 displayed on the display device 103.

As shown in FIG. 9, the display data 900 is displayed on the display screen of the display device 103. The display data 900 includes an image designation button 901 and an image display area 902. The image designation button 901 is a button for designating each form image on which the determination process is executed as a noteworthy image, and is displayed corresponding to each form image on which the determination process is executed. For example, each image designation button 901 is displayed side by side in the order of time when the determination process for the corresponding form image is executed. Among the image designation buttons 901, the button corresponding to the recognition error image is highlighted so as to be distinguishable from the buttons corresponding to other images.

In the image display area 902, the form image 903 designated as the image of interest by the image designation button 901 and the determination result in the determination process are displayed. In the image display area 902, a part or all of the form image 903 is displayed. In the image display area 902, the form image 903 is displayed so that the combination of the item values designated as the attention target is located at the reference position R. When the combinations of the item values designated as the objects of interest are arranged in the horizontal direction, the reference position R is set to, for example, a substantially central position in the vertical direction of the image display area 902. Further, when the combinations of the item values designated as the objects of interest are arranged in the vertical direction, the reference position R is set to, for example, a substantially central position in the horizontal direction of the image display area 902.

In the form image 903 shown in FIG. 9, in the determination process, "quantity x unit price = amount" is not established for "product E" and "product H", "total charge = total" is not established, and other relationships are not established. It is established.

When the item value of each specific item detected by the detection unit 122 does not have the relationship shown in the relationship information, the display control unit 124 sets the item value of the specific item to another specific item in the form image. Highlight it so that it can be distinguished from the item value of. When the item value of each specific item detected by the detection unit 122 does not have the relationship shown in the relationship information for the specific inspection target among the plurality of inspection targets, the display control unit 124 displays the form image. , Highlight the item value of each particular item corresponding to that particular inspection target. In the example shown in FIG. 9, since "quantity x unit price = amount" is not established for "product E" and "product H", the item values 906 corresponding to "product E" and "product H" correspond to each other. The item value 907 is highlighted. Further, since "total charge = total" is not established, the item value 904 corresponding to the "amount" and the item value 905 corresponding to the "total" are highlighted.

Further, when the relational information indicates a relationship between one item value of one specific item and a plurality of item values of another specific item, the display control unit 124 has one item of the one specific item. Highlight all values and multiple item values for other specific items. In the example shown in FIG. 9, since "total charge = total" is not established, each item value 904 corresponding to the "amount" and the item value 905 corresponding to the "total" are highlighted.

The display control unit 124 highlights the item value of each specific item by displaying it in a color different from the item value of the other specific item. The display control unit 124 may highlight the item value of each specific item by displaying it in a font or size different from the item value of the other specific item. As a result, the user can easily identify the item value for which the character recognition process is incorrect, and the inspection device 100 can improve the convenience of the user.

Further, the display control unit 124 has detected the specific item in the area adjacent to the area in which the item value of the specific item having no relationship shown in the relationship information is displayed in the form image by the detection unit 122. Display the item value. In the example shown in FIG. 9, the item values 904 and 909 detected by the detection unit 122 in the area adjacent to the area in which the item value 904 corresponding to the "amount" and the item value 905 corresponding to the "total" are displayed. Is displayed. Similarly, the item values 910 and 911 detected by the detection unit 122 are displayed in the area adjacent to the area where the item value 906 corresponding to "Product E" and the item value 907 corresponding to "Product H" are displayed. Will be done. As a result, the user can easily recognize which item value the detection result is incorrect, and the inspection device 100 can improve the convenience of the user.

Further, the display control unit 124 highlights the item value of the specific item for which the character recognition process is incorrect in the form image, or displays the item value detected by the detection unit 122 for the specific item. , The judgment result is superimposed and displayed on the form image. As a result, the user can visually recognize whether or not the character recognition process for each item value is correct, and the inspection device 100 can improve the convenience of the user.

Next, the display control unit 124 executes the reception process and ends a series of steps (step S207). The display control unit 124 changes the display data in response to an instruction from the user in the reception process. The details of the reception process will be described later.

FIG. 10 is a flowchart showing an example of the operation of the reception process. The reception process shown in FIG. 10 is executed in step S207 of the display process shown in FIG.

First, the display control unit 124 determines whether or not the first instruction using the input device 102 by the user has been accepted (step S301). The first instruction is an instruction to specify the image of interest to be displayed in the image display area 902. The display control unit 124 receives the first instruction to designate the form image corresponding to the image designation button 901 as the image of interest when any of the image designation buttons 901 is pressed by using a mouse, for example.

When the first instruction is received, the display control unit 124 sets the form image specified by the first instruction as the attention image (step S302). Next, the display control unit 124 identifies the recognition error target for the newly set attention image (step S303) and determines the attention target (step S304) in the same manner as in steps S204 and S205. Next, the display control unit 124 creates display data in the same manner as in step S206, displays it on the display device 103 (step S305), and returns the process to step S301. In this way, the display control unit 124 arbitrarily switches and displays the determination result for each form image in response to the first instruction by the user.

On the other hand, when the first instruction is not received, the display control unit 124 determines whether or not the second instruction using the input device 102 by the user has been accepted (step S306). The second instruction is an instruction to specify the attention image to be displayed in the image display area 902 from the recognition error images. The display control unit 124 receives the second instruction when, for example, the down arrow key or the up arrow key is pressed by using the keyboard while the image designation button 901 corresponding to the current attention image is designated. In that case, among the image designation buttons 901 corresponding to the recognition error image, the image designation button 901 located below or above the currently designated image designation button 901 and arranged closest to the image designation button 901. The corresponding form image is designated as the image of interest.

When the second instruction is received, the display control unit 124 sets the form image designated by the second instruction as the attention image (step S307). Next, the display control unit 124 identifies the recognition error target for the newly set attention image (step S303) and determines the attention target (step S304) in the same manner as in steps S204 and S205. Next, the display control unit 124 creates display data in the same manner as in step S206, displays it on the display device 103 (step S305), and returns the process to step S301. As described above, the display control unit 124 has a relationship in which the item values of each specific item detected by the detection unit 122 in each form image are shown in the relationship information in response to the second instruction by the user. No Judgment results for form images are sequentially switched and displayed. The user can sequentially select only the form images having an error in the character recognition process, and confirm or correct the error, and the inspection device 100 can improve the efficiency of the user's work. It will be possible.

On the other hand, when the second instruction is not received, the display control unit 124 determines whether or not the third instruction using the input device 102 by the user has been accepted (step S308). The third instruction is an instruction to specify an attention target from a plurality of inspection targets in the attention image displayed in the image display area 902. When any area in the image display area 902 is pressed by using a mouse, for example, the display control unit 124 receives a third instruction to specify a combination of item values corresponding to the pressed area as a target of interest. ..

When the third instruction is received, the display control unit 124 sets the combination of the item values specified in the third instruction as the attention target (step S309). Next, the display control unit 124 creates display data in the same manner as in step S206, displays it on the display device 103 (step S305), and returns the process to step S301. In this way, the display control unit 124 arbitrarily switches and displays the determination result for each inspection target in response to the third instruction by the user. The display control unit 124 can appropriately display the determination result for the inspection target that the user wants to pay attention to even when all the determination results for each inspection target cannot be collectively displayed in the image display area 902.

On the other hand, when the third instruction is not accepted, the display control unit 124 determines whether or not the fourth instruction using the input device 102 by the user has been accepted (step S310). The fourth instruction is an instruction to specify the attention target from the recognition error targets in the attention image displayed in the image display area 902. The display control unit 124 receives the fourth instruction when, for example, the down arrow key or the up arrow key is pressed by using the keyboard in a state where the combination of the item values corresponding to the current attention target is specified. In that case, among the combinations of item values corresponding to the recognition error target, the combination of item values located below or above the combination of item values corresponding to the current target of interest and arranged closest to the combination is related. The inspection target is designated as the attention target.

When the fourth instruction is received, the display control unit 124 sets the inspection target designated by the fourth instruction as the attention target (step S311). Next, the display control unit 124 creates display data in the same manner as in step S206, displays it on the display device 103 (step S305), and returns the process to step S301. In this way, the display control unit 124 has a relationship in which the item values of each specific item detected by the detection unit 122 among the inspection targets are shown in the relationship information in response to the fourth instruction by the user. The judgment results for the inspection target are sequentially switched and displayed. Even if the display control unit 124 cannot display all the judgment results for each inspection target collectively in the image display area 902, the display control unit 124 can appropriately select and display the judgment results for the inspection target having an error in the character recognition process. It will be possible. In particular, the user can sequentially select only the inspection target having an error in the character recognition process, and confirm or correct the error, and the inspection device 100 aims to improve the efficiency of the user's work. Is possible.

FIG. 11 is a schematic diagram showing an example of display data 1100 displayed on the display device 103.

FIG. 11 shows the display data 1100 displayed when the fourth instruction is received from the state in which the display data 900 shown in FIG. 9 is displayed on the display device 103. In the display data 900 shown in FIG. 9, the combination 906 of each item value related to the “commodity E” is designated as the object of interest and is located at the reference position R. On the other hand, in the display data 1100 shown in FIG. 11, the combination 907 of each item value related to the “commodity H” is newly designated as the object of interest. In the display data 1100, the form image 903 is scrolled and displayed so that the combination 907 of each item value related to the "product H" is located at the reference position R.

Therefore, the display control unit 124 first specifies the interval of each inspection target in the form image based on the detection result of the item value of the specific item. The display control unit 124 reads out the relative position of each item value of any specific item from the detection result table 600, and sets the distance between the relative positions of the item values adjacent to each other as the interval of each inspection target in the form image. Calculated as.

The display control unit 124 detects a ruled line extending between each item value from the form image by using a known image processing technique, and sets the distance between the adjacent ruled lines in the form image. It may be calculated as an interval for each inspection target.

Next, the display control unit 124 scrolls and displays the form image so that the newly designated attention target is displayed at the reference position R where the previously designated attention target is displayed. The display control unit 124 scrolls and displays the form image in the specified interval unit. For example, when the newly designated attention target is adjacent to the attention target designated in the past, the display control unit 124 scrolls and displays the form image by the specified interval. Further, for example, when there is one inspection target (expense item) between the newly designated attention target and the attention target designated in the past, the display control unit 124 has only twice the specified interval. Scroll the form image.

In this way, when the display control unit 124 newly accepts the designation of the attention target, the newly designated attention target and the attention target previously designated are displayed in the display area 902 of the form image. The form image is scrolled so that it is displayed at the position where it was. As a result, when the inspection target of interest is changed, the user can browse the newly designated inspection target without moving the line of sight, and the inspection device 100 improves the convenience of the user. It is possible to make it.

On the other hand, when the fourth instruction is not received, the display control unit 124 determines whether or not the correction instruction using the input device 102 by the user has been accepted (step S312). The correction instruction is an instruction to correct any item value among the item values designated as the object of interest to the correction value specified by the user. The display control unit 124 receives, for example, a correction instruction when a correction value is input using the keyboard in a state where a specific item value in the target of interest is specified.

Next, the display control unit 124 replaces the item value specified in the correction instruction with the correction value in the detection result table 600. The display control unit 124 refers to the relationship table 520 and specifies the relationship and the inspection target corresponding to the item value. Then, the display control unit 124 determines whether or not the error in the character recognition process has been corrected by determining whether or not the combination of the item values related to the specified inspection target has the specified relationship (step). S313). When there are a plurality of specified inspection targets, the display control unit 124 determines whether or not the error in the character recognition process has been corrected for each of the specified inspection targets.

When it is determined that none of the errors has been corrected, the display control unit 124 creates display data in the same manner as in step S206, displays it on the display device 103 (step S305), and returns the process to step S301.

On the other hand, when it is determined that any of the errors has been corrected, the display control unit 124 normally updates the determination result stored in the determination result table 610 in relation to the corrected error and the inspection target. (Step S314). Next, the display control unit 124 identifies the recognition error target for the current attention image (step S303) and determines the attention target (step S304) in the same manner as in steps S204 and S205. Next, the display control unit 124 creates display data in the same manner as in step S206, displays it on the display device 103 (step S305), and returns the process to step S301. If the item value that was highlighted due to an error in the character recognition process is corrected, the item value will not be highlighted, so the user can be sure that the correction has been made correctly. Become.

The display control unit 124 determines whether or not an error remains for each form image in which the error has been corrected, and processes the form image determined that no error remains by RPA processing via the communication device 101. It may be transmitted to a device (not shown). As a result, the RPA processing apparatus can automatically acquire only highly reliable form images and execute predetermined RPA processing.

On the other hand, when the correction instruction is not accepted, the display control unit 124 determines whether or not the end instruction using the input device 102 by the user has been accepted (step S315). If the display control unit 124 has not received the end instruction, the process returns to step S301, and if the end instruction is received, the display control unit 124 ends a series of steps.

As described in detail above, the inspection device 100 determines whether the item name candidate is included in the form image according to the priority order, and detects the item name candidate determined to be included first as a specific item. It is determined whether or not the detected item values of each specific item have a correct relationship. This makes it possible for the inspection device 100 to more accurately determine whether or not the item value is correctly detected from the form image.

Generally, a form such as a receipt contains a plurality of items, and each form has a wide variety of formats. From such a form, the item value of a specific item cannot be detected by the so-called standard OCR method in which the item value is detected from a predetermined area. On the other hand, the inspection device 100 determines whether the item name candidate is included in the form image according to the priority order, and detects the item name candidate determined to be included first as a specific item, so that the specific item is accurate. It became possible to identify it well.

Further, in the inspection device 100, it is possible to more accurately determine whether or not the item value is correctly detected from the form image, and it is possible to improve the reliability of the characters detected from the form image by using the OCR technique. It has become possible. Therefore, the inspection device 100 can reduce the work of confirming and correcting the detected characters by the user. In addition, since the detected characters are highly reliable, the user can use the detected characters for RPA technology to improve work efficiency.

FIG. 12 is a schematic diagram showing an example of another display data 1200 displayed on the display device 103.

In the display data 900 and 1100 shown in FIGS. 9 and 11, the item values detected by the detection unit 122 for the specific items having no relationship shown in the relationship information are displayed in the form image. On the other hand, in the display data 1200 shown in FIG. 12, only the item values of the specific items detected for the attention target are displayed in the form image. The item value of each specific item detected for the attention target is displayed in the area adjacent to the area where the item value of the specific item related to the attention target is displayed in the form image.

In this way, even if the display control unit 124 displays the item value of the specific item detected for the attention target in the area adjacent to the area where the item value of the specific item related to the attention target is displayed in the form image. good. As a result, the user can easily distinguish the inspection target of interest from another inspection target, and the inspection device 100 can improve the convenience of the user.

Further, in the display data 900 and 1100 shown in FIGS. 9 and 11, the image designation button corresponding to the form image in which the character recognition process was erroneous and the item value of the specific item were highlighted. On the other hand, in the display data 1200 shown in FIG. 12, only the image designation button corresponding to the attention image and the attention target are highlighted, and the image designation button corresponding to the form image other than the attention image and the inspection target other than the attention target are distinguished from each other. It is displayed as possible.

In this way, the display control unit 124 displays the image designation button corresponding to the attention image so as to be distinguishable from the image designation button corresponding to the form image other than the attention image, and can distinguish the attention target from the inspection target other than the attention target. It may be displayed in. As a result, the user can easily distinguish the form image and inspection target of interest from other form images and inspection targets, and the inspection device 100 can improve the convenience of the user. Become.

Further, when the arrow key is pressed by using the keyboard, the display control unit 124 does not select a new attention target from the recognition error targets, but a new inspection target adjacent to the current inspection target. It may be the object of attention. Even in that case, the display control unit 124 scrolls and displays the form image so that the newly designated attention target is displayed at the reference position where the attention target designated in the past was displayed.

FIG. 13 is a block diagram showing a schematic configuration of a processing circuit 130 in the inspection device 100 according to another embodiment.

The processing circuit 130 is used in place of the CPU 120, and executes determination processing and display processing in place of the CPU 120. The processing circuit 130 includes an image acquisition circuit 131, a detection circuit 132, a determination circuit 133, a display control circuit 134, and the like.

The image acquisition circuit 131 is an example of the image acquisition unit and has the same function as the image acquisition unit 121. The image acquisition circuit 131 acquires a form image from the image pickup device 104 and stores the acquired form image in the storage device 110.

The detection circuit 132 is an example of the detection unit, and has the same function as the detection unit 122. The detection circuit 132 reads a form image from the storage device 110, detects a specific item from the read form image, and stores the detection result in the storage device 110.

The determination circuit 133 is an example of the determination unit and has the same function as the determination unit 123. The determination circuit 133 reads the detection result from the storage device 110, determines whether or not the item values of each specific item have a predetermined relationship based on the read detection result, and stores the determination result in the storage device 110. ..

The display control circuit 134 is an example of the display control unit, and has the same function as the display control unit 124. The display control circuit 134 reads a form image, a detection result, and a determination result from the storage device 110, and displays display data on the display device 103 based on the read information. Further, the display control circuit 134 receives various instructions from the input device 102, changes the display data based on the received instructions, and updates the detection result and the determination result stored in the storage device 110.

As described in detail above, even when the inspection device 100 uses the processing circuit 130, the inspection device 100 can more accurately determine whether or not the item value is correctly detected from the form image.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments. For example, instead of using both "quantity x unit price = amount" and "total amount = billing amount" as the relationship between the item values of each specific item, only one of them or three or more relationships are used. It may be used.

Further, the form may include item values of a plurality of items, and the form may be a document other than an invoice such as a quotation, a purchase order, and a delivery note.

Further, the inspection device 100 may be a device that does not have an image pickup device, such as a PC (Personal Computer), a notebook PC, and a personal digital assistant. In that case, in step S101, the inspection device 100 communicates with an image pickup device such as a tablet PC, a multifunctional mobile phone, a scanner device, or a server device via the communication device 101, and receives a form image from each device. Get by doing. Further, even when the inspection device 100 is a device having an image pickup device such as a tablet PC or a multifunctional mobile phone, the inspection device 100 communicates with the scanner device or the server device via the communication device 101, and each form image is displayed. It may be acquired by receiving from the device.

100 Inspection device 110 Storage device 121 Image acquisition unit 122 Detection unit 123 Judgment unit 124 Display control unit

Claims (13)

Item name candidates are stored in order of priority for each of a plurality of element items, and the relationship between a numerical value that is one item value of one specific item and a numerical value that is a plurality of item values of another specific item is stored. A storage unit that stores the relational expression shown,
An image acquisition unit that acquires a form image that includes any of the item name candidates, and
For each of the plurality of element items, it is sequentially determined whether or not the corresponding item name candidate is included in the form image according to the priority order, and the item name candidate determined to be included first corresponds to the element item. The detector that detects as a specific item and
A determination unit for determining whether or not the item values of the plurality of specific items detected by the detection unit have the relationship shown in the relational expression.
It has a display control unit that displays a determination result by the determination unit.
When the item values of the plurality of specific items detected by the detection unit do not have the relationship shown in the relational expression, the display control unit has one item value of the one specific item and the other. Highlight all of the multiple item values for a particular item in
An inspection device characterized by that . The storage unit further stores a second relational expression showing the relationship between the item value of one specific item and the item value of two or more other specific items.
The inspection according to claim 1 , wherein the determination unit further determines whether or not the item values of the plurality of specific items detected by the detection unit have the relationship shown in the second relational expression. Device. The detection unit detects the item value of a specific item corresponding to each element item for each of a plurality of inspection targets.
In the display control unit, with respect to a specific inspection target among the plurality of inspection targets, the item values of the plurality of specific items detected by the detection unit do not have the relationship shown in the second relational expression. The inspection device according to claim 2 , wherein the item value of each specific item corresponding to the specific inspection target is highlighted. The inspection device according to claim 2 or 3 , wherein the display control unit displays the form image and displays the determination result superimposed on the form image. The detection unit detects the item value of a specific item corresponding to each element item for each of a plurality of inspection targets.
The inspection device according to claim 4 , wherein the display control unit receives a user's designation of a target of interest among the plurality of inspection targets. The display control unit displays the item value of the specific item detected for the attention target in the area adjacent to the area where the item value of the specific item related to the attention target is displayed in the form image. Item 5. The inspection device according to item 5. When the display control unit newly accepts the designation of the attention target, the position where the newly designated attention target is displayed in the display area of the form image and the attention target previously designated is displayed. The inspection device according to claim 5 or 6 , which scrolls and displays the form image so as to be displayed in. 7. The display control unit specifies an interval of each inspection target in the form image based on the detection result of the item value of the specific item, and scrolls and displays the form image in the specified interval unit. The inspection device described in. The inspection device according to any one of claims 5 to 8 , wherein the display control unit displays the attention target so as to be distinguishable from inspection targets other than the attention target. The display control unit
According to the first instruction by the user, the determination result for each of the plurality of form images is switched and displayed.
In response to the second instruction by the user, among the plurality of form images, the item values of the plurality of specific items detected by the detection unit are the same for the form image having no relationship shown in the relational expression. The inspection device according to any one of claims 1 to 9 , wherein the determination results are sequentially switched and displayed. The detection unit detects the item value of a specific item corresponding to each element item for each of a plurality of inspection targets.
The display control unit
In response to the third instruction by the user, the determination result for each of the plurality of inspection targets is switched and displayed.
According to the fourth instruction by the user, among the plurality of inspection targets, the item values of the plurality of specific items detected by the detection unit are the inspection targets having no relationship shown in the relational expression. The inspection device according to any one of claims 1 to 10 , wherein the determination results are sequentially switched and displayed. It is a control method of an inspection device having a storage unit.
Item name candidates are stored in the storage unit in order of priority for each of a plurality of element items , and between a numerical value that is one item value of one specific item and a numerical value that is a plurality of item values of another specific item. The relational expression showing the relation of is stored in the storage part, and
Obtain a form image that includes any of the item name candidates,
For each of the plurality of element items, it is sequentially determined whether or not the corresponding item name candidate is included in the form image according to the priority order, and the item name candidate determined to be included first corresponds to the element item. Detected as a specific item
It is determined whether or not the item value of each of the plurality of detected specific items has the relationship shown in the relational expression.
Including displaying the judgment result in the judgment.
In the above display, when the detected item values of the plurality of specific items do not have the relationship shown in the relational expression, one item value of the one specific item and a plurality of the other specific items. Highlight all the item values of
A control method characterized by that . A control program for an inspection device that has a storage unit.
Item name candidates are stored in the storage unit in order of priority for each of a plurality of element items , and between a numerical value that is one item value of one specific item and a numerical value that is a plurality of item values of another specific item. The relational expression showing the relation of is stored in the storage part, and
Obtain a form image that includes any of the item name candidates,
For each of the plurality of element items, it is sequentially determined whether or not the corresponding item name candidate is included in the form image according to the priority order, and the item name candidate determined to be included first corresponds to the element item. Detected as a specific item
It is determined whether or not the item value of each of the plurality of detected specific items has the relationship shown in the relational expression.
The inspection device is made to display the determination result in the determination.
In the above display, when the detected item values of the plurality of specific items do not have the relationship shown in the relational expression, one item value of the one specific item and a plurality of the other specific items. Highlight all the item values of
A control program characterized by that .

Images