JP2021125050A - Information processing apparatus - Google Patents

Abstract

To classify, based on an image of a receipt or a bill, the names and amounts of a number of purchases listed on the receipt or the bill.SOLUTION: In an information processing apparatus 10, a control unit 21 recognizes and extracts, from an image of a receipt or a bill picked up by an imaging camera 14, preset symbols and the names and amounts of purchases on rows where the symbols are printed, reads out divisions in charge associated with the extracted symbols from a storage unit 18, and classifies the extracted names and amounts of the purchases into the read-out divisions.SELECTED DRAWING: Figure 2

Description

The present invention relates to an information processing device including an imaging unit and a display unit for capturing an image, and more particularly to a technique for managing the amount of each item described in an accounting document such as a receipt or an invoice imaged by the imaging unit.

The information processing device includes an image pickup unit such as a camera and a display unit, and displays an image captured by the image pickup unit on the display unit. As such an information processing device, for example, there is a mobile phone used for various purposes.

In Patent Document 1, the bar code of the invoice is read by the camera of the mobile phone, the read bar code is decrypted by the mobile phone, and the decrypted data of this bar code is transmitted from the mobile phone to the AP server, and the AP server. Generates information for executing payment based on user information and barcode decryption data, sends the information for executing this payment from the AP server to the payment server, and executes the payment process by the payment server. The result is transmitted from the payment server to the AP server and the mobile phone.

Japanese Unexamined Patent Publication No. 2007-323613

Here, if many purchased item names and their amounts are described on the receipt or invoice and each purchased item name and their amount are to be classified, prior knowledge for this classification is required and the work is to be performed. Becomes complicated.

Therefore, if the information processing device can be used to classify the names of purchased items and their amounts described on the receipt or invoice, the time and effort of the user can be significantly reduced.

In Patent Document 1, the barcode of the invoice is read by the camera of the mobile phone and the payment process is executed based on the barcode. However, many purchased product names described in the receipt or the invoice and those are used. Does not classify the amount of money.

The present invention has been made in view of the above circumstances, and many of the images of accounting documents such as receipts or invoices are captured and described in the accounting documents based on the images of the captured accounting documents. The purpose is to make it possible to classify the names of purchased items and their amounts.

The information processing apparatus according to one aspect of the present invention corresponds to an imaging unit that captures an image of an accounting document in which an amount is described for each of a plurality of product names, a plurality of preset symbols, and each of the plurality of symbols. Recognize and extract the symbol and the purchased item name and amount of the line marked with the symbol from the storage unit that stores each assigned department and the image of the accounting document, and correspond to the extracted symbol. It is provided with a control unit that reads the assigned department in charge from the storage unit and classifies the extracted purchased product name and amount into the read department in charge.

According to the present invention, an image of an accounting document such as a receipt or an invoice is imaged, and many purchased item names and their amounts described in the accounting document are classified based on the image of the captured accounting document. can do.

It is a top view which shows the appearance of the information processing apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows the internal structure of the information processing apparatus which concerns on this embodiment. It is a figure which shows an example of a receipt. There is a flowchart showing a control procedure for classifying a plurality of purchased item names and their amounts described on a receipt. It is a figure which illustrates the initial screen displayed on the display part of an information processing apparatus. It is a figure which shows the message displayed on the screen of the display part. It is a figure which shows the management table which stored each department in charge, each symbol, and each keyword in association with each other. It is a figure which shows the result of the classification of the receipt based on a symbol for each department in charge. It is a figure which shows the result of the classification of the receipt based on a keyword for each department in charge. It is a figure which shows the result of the classification of the receipt by AI for each department in charge. It is a figure which aggregates and shows the result of classification by each department in charge. It is a figure which shows the address table which stored each charge department and each address in association with each other.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is a perspective view showing the appearance of the information processing apparatus according to the embodiment of the present invention. Further, FIG. 2 is a block diagram showing an internal configuration of the information processing apparatus according to the present embodiment.

The information processing device 10 of the present embodiment is, for example, a smartphone, a mobile device, or the like, and includes a display unit 11, a touch panel 12, a physical key 13, an image pickup camera 14, a network communication unit (NW communication unit) 15, and the like. It includes an image memory 16, a storage unit 18, and a control unit 19. These components are capable of transmitting and receiving data or signals to and from each other via the bus.

The display unit 11 is composed of a liquid crystal display (LCD: Liquid Crystal Display), an organic EL (OLED: Organic Light-Emitting Diode), and the like. For example, a plurality of icons are displayed on the screen of the display unit 11.

The touch panel 12 is a touch panel of a so-called resistance film type or capacitance type. The touch panel 12 is arranged on the screen of the display unit 11, detects the contact (touch) of the user's finger or the like with respect to the touch panel 12 together with the contact position, and inputs the user's instruction to the screen of the display unit 11. Therefore, the touch panel 12 serves as an operation unit for inputting a user operation on the screen of the display unit 11.

Further, the information processing device 10 includes a physical key 13 in addition to the above-mentioned touch panel 12 as an operation unit into which a user operation is input.

The image pickup camera 14 is a camera including a lens and an image pickup element (CCD, CMOS, etc.), captures an external image, and outputs image data indicating this image.

The network communication unit 15 is a communication interface including a communication module such as a LAN chip (not shown). The network communication unit 15 is connected to a terminal device such as an external server via a LAN, the Internet, or the like.

The image memory 16 stores image data or the like indicating an image captured by the image pickup camera 14.

The storage unit 18 is a storage device such as a RAM (Random Access Memory) and an HDD (Hard Disk Drive), and stores various application programs, various information, and the like.

The control unit 19 is composed of a processor, a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The processor is, for example, a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), an MPU (Micro Processing Unit), or the like. The control unit 19 functions as the control unit 21 when the control program stored in the ROM or the storage unit 18 is executed by the processor.

The control unit 19 is connected to a display unit 11, a touch panel 12, a physical key 13, an imaging camera 14, a network communication unit 15, an image memory 16, a storage unit 18, and the like, and controls the operation of these components and the operation of the components. Sends and receives signals or data to and from each component.

The control unit 21 serves as a processing unit that executes various processes. Further, the control unit 21 has a function of controlling the display operation of the display unit 11 and the communication operation of the network communication unit 15.

Here, the information processing apparatus 10 of the present embodiment is provided with an accounting document management function. The accounting document is, for example, a document in which the amount of money is described for each of a plurality of product names such as receipts, invoices, and quotations. Hereinafter, a receipt will be described as an example. When using the accounting document management function, the user looks at multiple purchased item names on the receipt and associates the purchased item name line that he / she knows in charge with in advance with the department in charge. Fill in the given symbol by hand. For example, in the receipt 31 shown in FIG. 3, each purchased item name S1 to S15 and each amount C1 to C15 are described in each line, and four types of symbols M1 to M4 are at the beginning of each purchased item name line ( It is written by hand in the blank). Each symbol M1 to M4 is associated with each department in charge B1 to B4 in advance.

The user operates the information processing device 10 to capture an image of the receipt 31 with the image pickup camera 14. The control unit 21 acquires image data indicating the image of the receipt 31 from the image pickup camera 14, stores the image data in the image memory 16, and displays the image of the receipt 31 on the display unit 11. Then, the control unit 21 recognizes each symbol M1 to M4, each purchased item name S1 to S15, and each amount C1 to C15 described in the receipt 31 based on the image of the receipt 31 by, for example, a known OCR technique. For the line marked with the symbol, the purchased item name and amount of the line are classified into the department in charge associated with the symbol. Further, the control unit 21 recognizes and extracts a preset keyword included in the line for the line not marked with the symbol, and associates the purchased item name and the amount of the line with the keyword in advance. Classify into the department in charge. Further, the control unit 21 estimates the department in charge from the character string included in the line by AI (artificial intelligence) and derives the line in which the symbol is not marked and does not include the keyword, and derives the line. Classify the purchased product name and amount of money into the department in charge derived by AI. Then, the control unit 21 collects the purchased product names or calculates the total amount for each department in charge B1 to B4, displays the result on the display unit 11, and notifies the department in charge through the network communication unit 15. ..

Next, the control procedure for classifying the plurality of purchased item names and their amounts described in the receipt as described above will be described in detail with reference to the flowchart shown in FIG.

It is assumed that the control unit 21 displays the initial screen G1 as shown in FIG. 5 on the display unit 11. On the initial screen G1, a plurality of icons 41a to 41d and the like associated with each function are displayed. When the user touches the icon 41d indicating the application program for executing the receipt management function, the control unit 26 detects the touch operation on the icon 41d through the touch panel 12 and stores the application program from the storage unit 18. It is read and activated, and as shown in FIG. 6, a message M1 indicating that the entry of a symbol on the receipt and the imaging of the receipt are urged is displayed on the display unit 11 (S101).

When the user sees the message M1 and finds the department in charge of managing the purchased product name described in the receipt, the user puts a symbol associated with the department in charge in the margin (for example, at the beginning) of the line of the purchased product name. Fill in by hand. Then, the user operates the physical key 13 of the information processing device 10 after the image of the receipt is captured by the imaging camera 14 and displayed on the screen of the display unit 11. When an image acquisition instruction is input to the physical key 13 by the operation, the control unit 21 acquires image data indicating the image of the receipt from the image pickup camera 14, stores the image data in the image memory 16, and stores the image data in the image memory 16. The image of the receipt 31 is displayed on the display unit 11 (S102). For example, an image of the receipt 31 as shown in FIG. 3 is captured, image data showing the image of the receipt 31 is stored in the image memory 16, and the image of the receipt 31 is displayed on the display unit 11.

The control unit 21 analyzes the image of the receipt 31 in the image memory 16 by a known OCR (Optical Character Recognition) function, and for each line of the receipt 31 in the image, the purchased item name described in the line, Characters indicating an amount of money or the like are recognized and extracted by, for example, OCR technology (S103).

Further, the control unit 21 cuts out the area of each character in the image of the receipt 31, extracts the character of each area (S104), and for each of the extracted characters, the character is any of the symbols M1 to M4. It is determined whether or not the characters match (S105). The control unit 21 makes the determination by, for example, a pattern matching technique. For example, if the characters do not match any of the symbols M1 to M4 (S105 "No"), the control unit 21 is printed to indicate the purchased item name, the amount of money, etc. described in the receipt 31. It is determined that the character is a character (S106).

Further, when the character matches any of the symbols M1 to M4 (S105 "Yes"), the control unit 21 determines the color of each pixel constituting the character in the image of the receipt 31, and the receipt 31. The color of each pixel constituting each other character around the character in the image is determined, and whether or not the character matching any of the symbols M1 to M4 and the color of each other character around the character are different. Is determined (S107).

For example, when the control unit 21 determines that the character matching any of the symbols M1 to M4 and the color of each other character around the character are the same (S107 "No"), the character is the receipt 31. It is determined that the characters are printed characters indicating the name of the purchased item, the amount of money, etc. described in (S106).

Further, when the control unit 21 determines that the character matching any of the symbols M1 to M4 and the color of each other character around the character are different (S107 "Yes"), the character is handwritten. It is determined that the character is a character (S108).

If a character matching any of the symbols M1 to M4 is a character included in the purchased item name, amount of money, etc. printed on the receipt 31, the color of that character and the other characters around the character are It will be the same. When the character corresponding to any of the symbols M1 to M4 is a handwritten character by a pen or the like, the color of the character and each other character around the character are different. Therefore, based on whether or not the character matching any of the symbols M1 to M4 and the color of each other character around the character are the same, the character matching any of the symbols M1 to M4 is handwritten. It is possible to determine whether or not the characters are entered in.

When the control unit 21 determines that the character matching any of the symbols M1 to M4 is a handwritten character (S108), the control unit 21 manages the characters stored in the storage unit 18 in advance as shown in FIG. 7. With reference to the table KT, the department in charge associated with the character (any of the symbols M1 to M4) is acquired (S109).

In the management table KT of FIG. 7, each symbol M1 to M4 and each keyword KW are associated and stored in association with each department in charge B1 to B4. The control unit 21 searches the management table KT for any of the symbols M1 to M4 determined to have been handwritten in S108, and reads and acquires the department in charge associated with the searched symbols.

When the control unit 21 acquires the department in charge associated with the handwritten symbol, the control unit 21 selects the line in which the handwritten symbol is written from each line of the receipt 31 extracted in S103, and selects this line. The name of the purchased item and the amount of money described in the line are classified into the department in charge of the acquisition (S110).

S105 to S110 are performed for all the symbols handwritten on the receipt 31, and the purchased item name and the amount of money written in the line where the symbols are handwritten are associated with the symbols. It is classified into departments.

For example, as shown in FIG. 8, the department in charge B1 is classified into the purchased item name S12 and the amount C12 in one line in which the symbol M1 is handwritten, and the department B2 in charge is handwritten with the symbol M2. The purchased item name S6 and amount C6 in the two lines, the purchased item name S9 and the amount C9 are further classified, and the purchased item name S11 and the amount C11 in one line in which the symbol M3 is handwritten are further classified in the department in charge B3. In addition, the department in charge B4 is classified into the purchased item name S3 and the amount C3 in one line in which the symbol M4 is handwritten.

Next, the control unit 21 selects a line in which the handwritten symbol is not entered from each line of the receipt 31 extracted in S103, and a character string (purchased item name, amount, etc.) included in the selected line. Is collated with each keyword KW in the management table KT of FIG. 7 (S111). Then, when the character string included in the selected line matches any of the keywords KW (S112 “Yes”), the control unit 21 refers to the management table KT of FIG. 7 and refers to the matched keyword KW. The department in charge associated with is read and acquired (S113).

When the control unit 21 acquires the department in charge associated with the keyword KW, the control unit 21 classifies the purchased item name and the amount of money described in the selected line into the acquired department in charge (S115).

S111 to S115 are performed for all lines in which the handwritten symbol is not entered in the receipt 31, and when the character string included in the line matches any keyword KW, the purchase item name and amount of the line are changed. , Classified into the department in charge associated with the keyword KW.

By the control unit 21, for example, as shown in FIG. 9, the department in charge B1 has three lines including the keyword KW "medicine", that is, the purchased product name S1 and the amount C1, the purchased product name S4 and the amount C4, the purchased product name S8 and the amount C8. Is classified, and since the keyword KW is not set in the department in charge B2, neither the purchased item name nor the amount of money is classified, and further, the department in charge B3 does not have a line containing the keyword KW "IT". None of the purchased product names and amounts are classified, and the department in charge B4 is classified into two lines of purchased product names S2 and amount C2 including the keyword KW "bag", and purchased product names S15 and amount C15.

Further, the control unit 21 selects a line in which the handwritten symbol is not entered and does not include the keyword KW from each line of the receipt 31 extracted in S103, and is included in the selected line. A character string (purchased item name, amount of money, etc.) is input to the AI built in the control unit 19, and the department in charge guided by this AI is acquired (S114). This AI is configured by using a known technique, and derives the department in charge from the character string included in the selected line based on the learning data generated in advance. For example, AI divides the character string "beef loin" contained in the line of the purchased product name S10 into the respective words "cow" and "loin", and based on these words and learning data, assigns the department B4 in charge. derive. Further, this AI is provided in a server or the like on the network, and the control unit 21 transmits the character string included in the selected line to the AI of the server through the network communication unit 15, and is in charge of being guided by this AI. The department may be received and acquired through the network communication unit 15. Further, if there is an error in the department in charge guided by AI, it is preferable to feed back the legitimate department in charge to AI so that the learning data can be updated.

When the control unit 21 acquires the department in charge guided by AI, the control unit 21 classifies the purchased item name and the amount of money described in the selected line into the acquired department in charge (S115).

In this way, S114 and S115 are performed for all lines in which the handwritten symbol in the receipt 31 is not entered and the keyword KW is not included, and the department in charge derives from the character string included in the line by AI. Then, the name of the purchased item and the amount of money described in the line are classified into the department in charge.

For example, as shown in FIG. 10, the department in charge B1 is not classified by AI for any purchased item name and amount, and the department in charge B2 is classified by AI for one line of purchased item name S7 and amount C7. Furthermore, the department in charge B3 is not classified by AI for any purchased product name and amount, and the department in charge B4 has four lines of purchased product name S5 and amount C5, purchased product name S10 and amount C10, purchased product name S13 and amount. C13, purchased product name S14, and amount C14 are classified.

When a handwritten symbol is marked on a line in this way, the purchased item name and amount of the line are classified into the department in charge associated with the symbol, and then the character string matching the keyword KW is placed on the line. If it is included, the purchased item name and amount of the line are classified into the department in charge associated with the keyword KW, and when the department in charge is derived from the character string included in the line by AI, the purchased item name of the line is derived. , The amount is categorized into the derived department.

Then, the control unit 21 generates a list IH that collectively shows the purchased item names, amounts, etc. classified into the departments in charge in S110 and S115 for each department in charge B1 to B4, and also generates the amount of money classified in the department in charge. The total amount SUM is calculated, and as shown in FIG. 11, the list IH and the total amount SUM are displayed on the screen of the display unit 11 for each department B1 to B4 in charge, and further, the creation date D of the list IH. Is displayed on the screen of the display unit 11 (S116). At this time, the user checks the list IH of each department in charge B1 to B4 displayed on the screen of the display unit 11 while looking at the receipt 31, and if there is an error, the user finds the error. Touch operation. When the control unit 21 detects the touch operation location through the touch panel 12, the control unit 21 displays a soft keyboard or the like on a part of the screen of the display unit 11, and the characters or characters input in response to the user's touch operation on the soft keyboard. Correct by changing the letters or numbers in the typographical error to the numbers, etc., and if this correction is a correction of the amount of the purchased item name, the total amount SUM of the department in charge to which the purchased item name is classified is also included. correct. Further, the touch panel 12 may accept an instruction to move an arbitrary line in response to a drag-and-drop operation on an arbitrary line displayed on the screen of the display unit 11 by the user. When the touch panel 12 receives the instruction, the control unit 21 moves an arbitrary line for which the drag-and-drop operation has been performed, and when the arbitrary line is moved from the original department in charge to another department in charge. Correct the total amount SUM of both departments in charge. After that, the control unit 21 stores the list IH and the creation date D of the respective departments B1 to B4 in the storage unit 18 as history information.

Further, the control unit 21 reads the addresses AD1 to AD4 of the respective departments B1 to B4 from the address table AT as shown in FIG. 12 stored in advance in the storage unit 18, and is in charge of each department B1 to B4. Using the address of the department, the purchased item name, amount, total amount SUM, and creation date D classified into the department in charge are transmitted to the department in charge through the network communication unit 15 (S117).

As described above, in the present embodiment, the purchased product name and amount of each line are distributed to the respective departments B1 to B4 based on each symbol M1 to M4 handwritten on the receipt 31 or each keyword KW included in each line. Furthermore, since the purchased item name and the amount of money of each line are sorted and classified by the departments in charge B1 to B4 by AI, the time and effort of the user can be significantly reduced.

In the above embodiment, the receipt is illustrated, but the present invention is also applied to the invoice, the quotation, etc., and the purchased product name, the amount, etc. described in the invoice, the quotation, etc. are used. It is also possible to classify into the department in charge.

Further, although the mobile phone is illustrated in the above embodiment, the present invention can be applied to other types of electronic devices including an imaging unit and a display unit. For example, some image forming devices include an image reading unit for reading an image of a document and a display unit, and if this image reading unit is applied as an imaging unit, the image reading unit reads the image of the receipt and the receipt. Image can be displayed on the display unit. That is, the image forming apparatus can also be an example of the information processing apparatus according to the present invention.

Further, the configuration of the above-described embodiment described with reference to FIGS. 1 to 12 is merely an example of the present invention, and is not intended to limit the present invention to the configuration.

10 Information processing device 11 Display unit 12 Touch panel 13 Physical key 14 Imaging camera 15 Network communication unit 16 Image memory 18 Storage unit 19 Control unit 21 Control unit

Claims (8)

An imaging unit that captures images of accounting documents in which the amount of money is listed for each of multiple product names,
A storage unit that stores a plurality of preset symbols and their respective departments associated with each of the plurality of symbols.
The symbol and the purchased item name and amount of money in the line marked with the symbol are recognized and extracted from the image of the accounting document, and the department in charge associated with the extracted symbol is read from the storage unit to read the above. An information processing device including a control unit that classifies the extracted purchased product name and amount into the read department in charge. The control unit determines whether or not the symbol recognized and extracted from the image of the accounting document is hand-written, and if it is determined that the symbol is hand-written, the extraction is performed. The information processing device according to claim 1, wherein the department in charge associated with the symbol is read from the storage unit, and the extracted purchased product name and amount are classified into the read department in charge. The information processing device according to claim 1 or 2, wherein the control unit calculates the total amount of money classified into the department in charge. The destination information of each department in charge is stored in the storage unit.
The control unit classifies the extracted purchased product name and amount into the read department in charge, reads the destination information of the department in charge from the storage unit, and reads the destination information of the read department in charge based on the destination information of the department in charge. The information processing apparatus according to any one of claims 1 to 3, which performs a process for notifying the destination indicated by the destination information of the department in charge of the purchased product name and the amount of money classified in the above. Keywords associated with the respective departments in charge are stored in the storage unit.
The control unit recognizes and extracts the keyword included in the line and the purchased item name and amount of the line for the line not marked with the symbol, and the department in charge associated with the extracted keyword. The information processing apparatus according to any one of claims 1 to 4, wherein the information processing device is read from the storage unit and the extracted purchased product name and amount are classified into the read department in charge. The control unit recognizes and extracts the character string included in the line and the purchased product name and amount of the line for the line not marked with the symbol, and the extracted character string and preset learning data. The information processing apparatus according to any one of claims 1 to 4, wherein the department in charge is estimated based on the above, and the extracted purchased product name and amount are classified into the estimated department in charge. A display unit for displaying an image of the accounting document captured by the imaging unit is further provided.
The information processing device according to any one of claims 1 to 6, wherein the control unit displays the result of the classification on the display unit. The information processing device according to any one of claims 1 to 7, wherein the control unit stores the purchased item name and the amount of money in the department in charge and the line classified into the department in charge as history information in the storage unit.

Images