JP2020021458A - Information processing apparatus, information processing method, and information processing system - Google Patents

Abstract

To make it possible to perform electronic settlement of a bill not having a bar code.SOLUTION: An information terminal device 10 comprises: an image acquisition unit 22 that acquires a bill image; an identification information extraction unit 24 that extracts identification information indicating a bill source from the bill image; a type determination unit 25 that determines the type of the bill corresponding to the identification information; an area specification unit 26 that specifies an area of the bill image in which a character image related to electronic settlement is included on the basis of a format corresponding to the type of the bill determined by the type determination unit 25; a character recognition unit 27 that recognizes characters in the area; a settlement information extraction unit 28 that extracts characters related to the electronic settlement from the area; a display control unit 29 that displays the characters related to the electronic settlement on a display unit 12; and a communication unit 14 that, when a touch panel 13 receives a settlement instruction, transmits the characters related to the electronic settlement to a settlement server 30.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an information processing apparatus, an information processing method, and an information processing system, and in particular, performs image processing on an acquired invoice image to extract information related to electronic payment, output the information to a payment server, and perform electronic payment at the payment server. Related to the technology to be executed.

  Patent Literature 1 discloses that in a system including a mobile phone, an application server, and a payment server, the mobile phone reads and decodes a barcode of a bill, transmits barcode decryption data to the application server, and the application server transmits the barcode. Disclosure of electronic payment of credit cards by generating information for executing payment from code decoding data and transmitting it to the payment server, and returning the payment processing result at the payment server to the mobile phone and application server Have been.

JP 2007-323613 A

  However, in the system described in Patent Literature 1, it is necessary to read the barcode of a bill with a barcode, and electronic payment cannot be performed for a bill without a barcode.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to enable electronic payment of a bill without a barcode.

  An information processing apparatus according to one aspect of the present invention includes an image acquisition unit that acquires a bill image, which is an image of a bill for a user, and extracts identification information indicating a billing source from the bill image acquired by the image acquisition unit. Identification information extraction unit, a type identification unit that identifies the type of invoice corresponding to the identification information extracted by the identification information extraction unit, and a format storage unit that stores in advance the format of each invoice for each type of invoice And reading out a format corresponding to the type of invoice determined by the type determination unit from the format storage unit, and based on the read out format of the invoice, an area where the invoice image includes a character image related to electronic payment. A character recognition unit that recognizes a character in the region specified by the region specification unit; a display unit; and an electronic device among characters in the region recognized by the character recognition unit. A payment information extracting unit that extracts characters related to the electronic payment, a display control unit that causes the display unit to display characters related to the electronic payment extracted by the payment information extracting unit, and a payment processing unit that displays the contents of the electronic payment displayed on the display unit. An operation unit for receiving an instruction from a user; and a communication unit for transmitting, when the operation unit receives the payment instruction, characters related to the electronic payment to a payment server.

  Further, an information processing method according to another aspect of the present invention includes an image acquiring step of acquiring a bill image, which is an image of a bill for a user, and a billing source from the bill image acquired in the image acquiring step. An identification information extraction step of extracting identification information to be indicated, a type determination step of determining a type of invoice corresponding to the identification information extracted in the identification information extraction step, and a format of each invoice for each type of invoice. A format corresponding to the bill type determined in the type determination step is read from a format storage unit stored in advance, and a character image related to electronic payment is included in the bill image based on the read bill format. An area identifying step of identifying an area to be identified; a character recognizing step of recognizing a character in the area identified in the area identifying step; A payment information extraction step of extracting characters related to electronic payment among the characters in the area recognized in the step, and a display control step of displaying characters related to electronic payment extracted in the payment information extraction step on a display unit; An operation step of receiving from the user a payment instruction regarding the content of the electronic payment displayed on the display unit, and transmitting the character related to the electronic payment to a payment server when the payment instruction is received in the operation step. And a communication step.

  An information processing system according to yet another aspect of the present invention is an information processing system in which an information processing device and a settlement server can communicate via a network, and the information processing device acquires a bill image. An image acquisition unit that acquires a bill image that is an image of a bill for a user; and an identification information extraction unit that extracts identification information indicating a billing source from the bill image acquired by the image acquisition unit. A communication unit for transmitting the identification information extracted by the identification information extraction unit to the payment server, wherein the payment server corresponds to the identification information received from the communication unit and extracted by the identification information extraction unit. A type discriminating unit for discriminating the type of invoice, a format storage unit for preliminarily storing the format of each invoice for each type of invoice, From the format storage unit, based on the format of the read invoice, an area specifying unit that specifies an area including a character image related to electronic payment in the invoice image, and the area specified by the area specifying unit. A server communication unit that transmits to the information processing device, the information processing device, the communication unit recognizes a character in the area received from the server communication unit character recognition unit, a display unit, A payment information extraction unit that extracts a character related to electronic payment among characters in the area in which the character recognition unit has recognized the character, and a display control unit that causes the display unit to display the character related to electronic payment extracted by the payment information extraction unit. An operation unit that accepts a payment instruction from a user for the content of the electronic payment displayed on the display unit, wherein the communication unit includes the operation unit When the payment instruction is received, a character related to the electronic payment is transmitted to the payment server, and the payment server receives an electronic payment based on the character information related to the electronic payment from the communication unit received by the server communication unit. Make payments.

  ADVANTAGE OF THE INVENTION According to this invention, the electronic payment of the bill which does not have a barcode can be performed. In addition, since characters related to electronic payment can be generated and transmitted from various invoice images, electronic payment can be performed for various invoices, and convenience can be improved.

1 is a front view of an information terminal device as an example of an information processing device according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating an internal configuration of the information terminal device in the information processing system according to the first embodiment. It is a figure showing an example of a correspondence storage table of identification information and the kind of bill. It is a flowchart which shows an example of the electronic payment processing of 1st Embodiment. FIG. 7A is a diagram illustrating an example of a bill image, and FIG. 7B is a diagram illustrating an example of an information terminal device in which extracted characters related to electronic payment are displayed on a display unit. (A) is a diagram showing an example of a character image related to electronic payment in a bill image, and (B) is a diagram showing an example of extracting an identification image in a bill image. It is a block diagram showing an internal configuration of an information terminal device and a payment server in an information processing system of a second embodiment.

  Hereinafter, an information processing apparatus and an information processing system according to an embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a front view of an information terminal device which is an example of an information processing device according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating an internal configuration of the information terminal device in the information processing system according to the first embodiment.

  The information processing system 1 according to the first embodiment includes an information terminal device 10 and a payment server 30. The information terminal device 10 and the settlement server 30 are communicably connected to each other via a network.

  Note that an electronic payment application provided by the payment server 30 is installed in the information terminal device 10. By executing the electronic payment application, the information terminal device 10 has a function of generating characters related to the electronic payment from the bill image and transmitting the characters to the payment server 30, a function of receiving the result of the electronic payment from the payment server 30, and the like. Is done.

  The payment server 30 is a server that provides an electronic payment application to the information terminal device 10, provides an electronic payment service to the information terminal device 10 that executes the electronic payment application, and performs electronic payment. For example, the settlement server 30 executes settlement by bank / postal savings, settlement by credit card, settlement by electronic money, and the like.

  The information terminal device 10 is, for example, a portable terminal device such as a tablet computer, and includes a control unit 11, a display unit 12, a touch panel 13, a communication unit 14, a storage unit 15, and an imaging unit 16. ing. These components can transmit and receive data or signals to and from each other via a bus.

  The display unit 12 includes a liquid crystal display (LCD: Liquid Crystal Display), an organic EL (OLED: Organic Light-Emitting Diode), and the like.

  The touch panel 13 is a touch panel of a so-called resistive type or a capacitive type. The touch panel 13 is arranged on the screen of the display unit 12 and detects a touch of a finger or the like on the screen of the display unit 12 together with the contact position. When the touch panel 13 detects contact with a finger or the like, the touch panel 13 outputs a detection signal indicating coordinates of the contact position to the control unit 21 of the control unit 11 or the like. Therefore, the touch panel 13 functions as an operation unit to which a user operation on the screen of the display unit 12 is input.

  Further, the information terminal device 10 may include hard keys in addition to the touch panel 13 as an operation unit to which a user operation is input.

  The communication unit 14 is a communication interface including a communication module such as a LAN chip (not shown). The communication unit 14 has various communication functions such as a communication function by e-mail and a communication function with the settlement server 30. The communication unit 14 can receive, for example, a bill image in PDF (Portable Document Format) attached to an e-mail.

  The storage unit 15 includes a large-capacity solid state drive (SSD), a hard disk drive (HDD), and the like, and stores various data and programs. The storage unit 15 includes a user information storage unit 151, an identification information storage unit 152, and a format storage unit 153.

  The user information storage unit 151 stores in advance authentication information of the user and personal information of the user in association with each other. The user authentication information is composed of a user ID (login name) as a user-specific identification code required to log in as an authorized person of the electronic payment application started by the information terminal device 10 and a password. Is done. Further, the personal information of the user is personal information registered in advance by the user, and includes characters of a bank name (for example, XYZ Bank) where the user has an account, characters of an account holder, characters of an account number, characters of the user, The characters of the mobile phone number are included.

  The identification information storage unit 152 stores in advance identification information indicating a billing source such as a bank or a credit company. For example, as shown in FIG. 3, the identification information storage unit 152 stores the logos L1, L2, L3, L4, etc. of the respective companies as the identification information indicating the billing source, and also stores the logos L1, L2, L2, L3 and L4 are stored in association with the company that is the billing source indicated by each logo mark. For example, the logo mark L1 indicates a company A (specifically, the company A is an ABC Bank), and the logo mark L2 indicates a company B.

  For example, as shown in FIG. 5B described later, the logo mark L1 has a company name (here, the name of ABC bank) as a company name logo L11 using ordinary characters, and a symbol mark L12 that symbolizes the company (here, ABC bank). including. Note that the identification information is not limited to a logo mark, and may include at least one of a symbol mark, a company name logo, and a symbol logo with characters in which the company name is designed.

  The form storage unit 153 stores in advance the form of each bill for each bill type. For example, the type of bill differs depending on the company from which the bill is issued. As shown in FIG. 3, the form storage unit 153 stores the form of the bill for each company that is the billing source. The form storage unit 153 stores a form A1 of a bill of Company A (ABC Bank), a form B1 of a bill of Company B, a form C1 of a bill of Company C, and a form D1 of a bill of Company D. .

  The photographing unit 16 is, for example, a camera. The photographing unit 16 photographs a bill for the user according to an operation by the user.

  The control unit 11 includes a processor, a random access memory (RAM), a read only memory (ROM), and the like. The processor is a CPU (Central Processing Unit), an MPU, an ASIC, or the like. When the control program stored in the ROM or the storage unit 15 is executed by the processor, the control unit 11 controls the control unit 21, the image acquisition unit 22, the user authentication unit 23, the identification information extraction unit 24, It functions as a determination unit 25, an area identification unit 26, a character recognition unit 27, a payment information extraction unit 28, and a display control unit 29. Note that each of the above-described components of the control unit 11 may be configured by a hardware circuit without depending on the operation based on the above-described control program.

  The control unit 21 controls the overall operation of the information terminal device 10. The control unit 21 is connected to the display unit 12, the touch panel 13, the communication unit 14, the storage unit 15, the imaging unit 16, and the like, and controls the operation of each of the above-described components, and transmits signals to and from each of the components. Alternatively, data is transmitted and received.

  Further, the control unit 21 specifies a user operation input by the user based on the detection signal output from the touch panel 13. For example, the user operation includes a touch operation (tap operation). And the control part 21 performs control according to the specified user operation. For example, when the user performs a touch operation on an icon (image display) 121 indicating the electronic payment application displayed on the screen of the display unit 12 illustrated in FIG. And a detection signal indicating a touch operation is output to the control unit 21, and the control unit 21 activates the electronic payment application based on the detection signal. By performing an operation or the like in accordance with the electronic payment application, it is possible to generate and transmit characters related to electronic payment from the bill image. Details will be described later.

  The user authentication unit 23 determines whether the input information for user authentication input to the touch panel 13 matches the authentication information stored in the user information storage unit 151. Specifically, the display control unit 29 displays a screen prompting the user to input a user ID (login name) and a password when the electronic payment application executed by the control unit 21 is started. And the information terminal device 10 is in a state of waiting for input of input information. When the input information for user authentication is input to the touch panel 13, the user authentication unit 23 determines whether the input information matches the authentication information stored in the user information storage unit 151.

  The image acquisition unit 22 acquires a bill image, which is an image of a bill for the user. If the user authentication unit 23 determines that the passwords match, the image acquisition unit 22 acquires the bill image captured by the capturing unit 16 or the bill image attached to the mail received by the communication unit 14. .

  Specifically, when the user authentication unit 23 determines that the images match, the image capturing unit 16 can capture a bill for the user. Then, the photographing unit 16 photographs a bill for the user according to an operation by the user. Then, the image acquiring unit 22 acquires the bill image photographed by the photographing unit 16 and stores the bill image in the storage unit 15. Further, when the user authentication unit 23 determines that there is a match, the image acquisition unit 22 acquires the invoice image if the invoice image is attached to the mail received by the communication unit 14, and stores the invoice image in the storage unit 15. To memorize.

  The identification information extracting unit 24 extracts identification information indicating a billing source from the bill image acquired by the image acquiring unit 22. For example, as illustrated in FIG. 2, the identification information extraction unit 24 analyzes the bill image acquired by the image acquisition unit 22 in a layout analysis, and indicates the identification information in the bill image analyzed by the analysis unit 241. An identification image area specifying unit 242 for specifying an identification image area including the identification image. The identification information extraction unit 24 extracts the identification information indicated by the identification image by performing image recognition or character recognition on the identification image region specified by the identification image region specifying unit 242.

  The analysis unit 241 analyzes the layout of the bill image G1 shown in FIG. The identification image area specifying unit 242 includes identification images (company name logo L11 and symbol mark L12 constituting the logo L1) in the bill image shown in FIG. 5B analyzed by the analysis unit 241 in the layout. The identification image area T1 is specified. Then, the identification information extraction unit 24 performs image recognition or character recognition on the identification image area T1 specified by the identification image area specifying unit 242, and thereby the identification information (ABC Bank L11 and the symbol mark L12) indicated by the identification image (ABC Bank). , Its symbol mark). In the present embodiment, the company name logo L11 and the symbol mark L12 are all extracted. However, since these indicate the same company, any one of them may be extracted. Even in this case, the billing source of the bill, that is, the company can be specified.

  The type determination unit 25 determines the type of the bill corresponding to the identification information extracted by the identification information extraction unit 24. As shown in FIG. 3, the correspondence information storage table stores identification information, a company name, and a bill type in association with each other in advance. Specifically, when the identification information extracted by the identification information extracting unit 24 is the logo mark L1 using the correspondence storage table shown in FIG. (ABC Bank) and that the type of invoice corresponding to Company A (ABC Bank) is in the form A1.

  The area specifying unit 26 reads out a format corresponding to the type of invoice determined by the type determination unit 25 from the format storage unit 153, and based on the read out format of the invoice, a character image related to electronic payment is included in the invoice image. Identify the included area. For example, when the format corresponding to the type of invoice determined by the type determination unit 25 is the format A1 shown in FIG. 3, the area specifying unit 26 stores the format A1 of the invoice of Company A (ABC Bank) in a format. Read from the unit 153. Then, the area specifying unit 26 specifies an area T2 in which the character image related to the electronic payment is included in the invoice image G1 as shown in FIG. 6A described later, based on the format A1 of the read invoice. The format A1 includes the arrangement information of the area T2 in the bill image G1. Therefore, the area specifying unit 26 specifies the area T2 in the invoice image G1 using the arrangement information included in the invoice form A1.

  The character recognizing unit 27 recognizes a character in the area specified by the area specifying unit 26 (that is, an area including a character image related to electronic payment). For example, the character recognition unit 27 recognizes characters in the area T2 by performing a known OCR (optical character recognition) process on the area T2 illustrated in FIG.

  The payment information extraction unit 28 extracts characters related to electronic payment from the characters in the area recognized by the character recognition unit 27. For example, as shown in FIG. 6 (A), the payment information extracting unit 28 determines, among the characters in the area T2 that the character recognizing unit 27 has recognized the character, as a character relating to electronic payment, a bank (for which the billing source has an account). In other words, a letter T21 indicating the account holder and account number of the (payee), a letter T22 indicating the name of the due date and the due date, a letter T23 indicating the name and amount of the total payment, and the minimum amount of the divided payment which is an example of the divided payment The name of the sum and the character T24 indicating the sum are extracted.

  In addition, the payment information extraction unit 28 determines, from the personal information of the user stored in the user information storage unit 151, at least the character P10 of the name of the bank (for example, XYZ bank) where the user has an account, the character P11 of the account holder, and the like. The character P13 of the account number is obtained. In the present embodiment, the payment information extraction unit 28 further acquires the character P12 of the mobile phone number of the user from the personal information of the user stored in the user information storage unit 151.

  The display control unit 29 controls the display operation of the display unit 12 under the control of the control unit 21. In addition, the display control unit 29 causes the display unit 12 to display characters related to the electronic payment extracted by the payment information extraction unit 28. For example, as shown in FIG. 6 (B), the display control unit 29 sets the characters T21 indicating the account holder and account number of the bank where the billing source has the account as the characters related to the electronic payment extracted by the payment information extracting unit 28. The display unit 12 displays the name of the due date and the character T22 indicating the due date, the name of the total payment amount and the character T23 indicating the amount, and the name of the minimum payment amount and the character T24 indicating the amount thereof. Further, as shown in FIG. 6 (B), the display control unit 29 stores, as the personal information of the user stored in the user information storage unit 151, the character P10 of the bank name of the user acquired by the payment information extracting unit 28. The character P11 of the account holder of the user, the character P12 of the mobile phone number of the user, and the character P13 of the account number of the user are displayed on the display unit 12.

  The touch panel 13 receives a payment instruction from the user regarding the content of the electronic payment displayed on the display unit 12.

  The communication unit 14 transmits characters related to the electronic payment to the payment server 30 when the touch panel 13 receives the payment instruction.

  The payment server 30 performs electronic payment based on the received characters related to electronic payment. For example, the payment server 30 uses the characters P10 of the name of the bank (for example, XYZ Bank) in which the user has an account, the characters P11 of the account holder, and the characters P13 of the account number among the characters related to the electronic payment received, Specify the name, account holder, and account number. Further, the settlement server 30 specifies the name of the payee, the account holder of the payee and the account number of the payee from the character T21 indicating the account holder and account number of the bank having the account, and calculates the total amount of the payment. The payment amount is specified from the name T23 indicating the name and the amount thereof, or the character T24 indicating the name of the minimum installment payment amount and the amount thereof, and the date for performing the electronic settlement is determined from the name of the payment date and the character T22 indicating the payment date. Check that it is within the day. The payment server 30 performs an electronic payment in which a payment amount designated by the user is withdrawn from a bank (for example, XYZ Bank) in which the user has an account, and the withdrawn payment amount is paid to a billing source (for example, ABC Bank). . Then, the payment server 30 transmits the result of the electronic payment to the communication unit 14 of the information terminal device 10. The display control unit 29 of the information terminal device 10 causes the display unit 12 to display the result of the electronic payment received by the communication unit 14 under the control of the control unit 21.

  Next, an electronic payment process in which the information terminal device 10 of the first embodiment generates and transmits characters related to electronic payment from the acquired user's bill image will be described with reference to the flowchart shown in FIG.

  First, in the information terminal device 10, when the user performs a touch operation on the icon 121 indicating the electronic payment application on the display unit 12 illustrated in FIG. 1, the operation is detected by the touch function of the touch panel 13, and the control unit 21 The electronic payment application is activated based on the detection signal from the touch panel 13 (S11).

  The display control unit 29 causes the display unit 12 to display a screen prompting the user to input a user ID (login name) and a password when the electronic payment application executed by the control unit 21 is started. The terminal device 10 waits for input of input information. When the user ID (login name) and password are input to the touch panel 13 as input information for user authentication, the user authentication unit 23 outputs the input user ID (login name) and password, and the user information storage unit 151. It is determined whether or not the regular user ID (login name) and the password, which are the authentication information stored in, match (S12: user authentication step).

  If the user authentication unit 23 determines that they do not match (S12 “No”), the control unit 21 ends this processing. That is, since the login of the user is not confirmed, the control unit 21 terminates the electronic settlement processing without completing the electronic settlement processing without acquiring the bill image.

  On the other hand, when the user authentication unit 23 determines that they match (S12 “Yes”), the image acquisition unit 22 acquires a bill image (S13: image acquisition step). For example, when the user authentication unit 23 determines that they match (S12 “Yes”), the image capturing unit 16 can capture an invoice for the user. Then, the photographing unit 16 photographs a bill for the user according to an operation by the user. Then, the image acquiring unit 22 acquires the bill image photographed by the photographing unit 16 and stores the bill image in the storage unit 15. Here, it is assumed that the image acquisition unit 22 has acquired the bill image G1 shown in FIG. In addition to the acquisition of the bill image by the photographing unit 16, the acquisition of the bill image by the communication unit 14 is also possible. For example, if the user authentication unit 23 determines that there is a match, the image acquisition unit 22 acquires the invoice image if the invoice image is attached to the mail received by the communication unit 14.

  The identification information extracting unit 24 extracts identification information indicating a billing source from the bill image acquired by the image acquiring unit 22 (S14: identification information extracting step). Specifically, the analysis unit 241 of the identification information extraction unit 24 performs a layout analysis on the bill image G1 shown in FIG. The identification image area identification section 242 of the identification information extraction section 24 includes identification images (company name logo L11 and symbol mark L12) indicating identification information in the bill image shown in FIG. The identification image area T1 is specified. Then, the identification information extraction unit 24 performs image recognition or character recognition on the identification image area T1 specified by the identification image area specifying unit 242, and thereby the identification information (ABC Bank L11 and the symbol mark L12) indicated by the identification image (ABC Bank). , Its symbol mark).

  For example, as illustrated in FIG. 5B, when the identification information extraction unit 24 extracts the identification information (S14 “Yes”), the type determination unit 25 corresponds to the identification information extracted by the identification information extraction unit 24. The type of bill is determined (S15: type determination step). Specifically, the type discriminating unit 25 uses the correspondence storage table shown in FIG. 3 to identify the identification information extracted by the identification information extracting unit 24 as a logo L1. ), It is determined that the type of invoice corresponding to Company A (ABC Bank) is in the form A1.

  On the other hand, if the identification information extraction unit 24 fails to extract the identification information in S14 (“No” in S14), the control unit 21 ends this processing. In other words, when the identification information cannot be extracted by the identification information extracting unit 24, the control unit 21 terminates the electronic settlement processing without completing the electronic settlement processing because the type of bill cannot be determined.

  The area specifying unit 26 reads out a format corresponding to the type of invoice determined by the type determination unit 25 from the format storage unit 153, and based on the read out format of the invoice, a character image related to electronic payment is included in the invoice image. The included area is specified (S16: area specifying step). For example, when the format corresponding to the type of invoice determined by the type determination unit 25 is the format A1 shown in FIG. 3, the area specifying unit 26 stores the format A1 of the invoice of Company A (ABC Bank) in a format. Read from the unit 153. Then, the area specifying unit 26 specifies an area T2 in which the character image related to the electronic payment is included in the invoice image G1 as shown in FIG. 6A described later, based on the format A1 of the read invoice.

  The character recognizing unit 27 recognizes characters in the area specified by the area specifying unit 26 (that is, an area including a character image related to electronic payment) (S17: character recognition step). For example, the character recognition unit 27 recognizes characters in the area T2 by performing a known OCR (optical character recognition) process on the area T2 illustrated in FIG.

  The payment information extraction unit 28 extracts a character related to electronic payment from the characters in the area recognized by the character recognition unit 27 (S18: payment information extraction step). For example, as shown in FIG. 6A, the payment information extracting unit 28 determines, among the characters in the area T2 recognized by the character recognizing unit 27, characters of the bank in which the billing source has an account, as characters relating to electronic payment. A character T21 indicating the account holder and account number, a character T22 indicating the name of the due date and its due date, a character T23 indicating the name of the total payment amount and its amount, a character T24 indicating the name of the minimum payment amount and the amount thereof, respectively Extract. Further, the payment information extracting unit 28 extracts the character P10 of the name of the bank (for example, XYZ bank) in which the user has an account, the character P11 of the account holder, the character P11 from the personal information of the user stored in the user information storage unit 151. And the character P13 of the user's account number.

  The display control unit 29 causes the display unit 12 to display the characters related to the electronic payment extracted by the payment information extraction unit 28 (S19: display control step). For example, as shown in FIG. 6 (B), the display control unit 29 sets the characters T21 indicating the account holder and account number of the bank where the billing source has the account as the characters related to the electronic payment extracted by the payment information extracting unit 28. The display unit 12 displays the name of the due date and the character T22 indicating the due date, the name of the total payment amount and the character T23 indicating the amount, and the name of the minimum payment amount and the character T24 indicating the amount thereof. Further, as shown in FIG. 6 (B), the display control unit 29 stores, as the personal information of the user stored in the user information storage unit 151, the character P10 of the bank name of the user acquired by the payment information extracting unit 28. The character P11 of the account holder, the character P12 of the mobile phone number of the user, and the character P13 of the account number of the user are displayed on the display unit 12.

  The control unit 21 determines whether there is an electronic payment instruction (S20). Specifically, the control unit 21 determines whether the touch panel 13 has received a payment instruction from the user regarding the content of the electronic payment displayed on the display unit 12. When the user touches the button K1 indicating “collective payment” or the button K2 indicating “installment payment” illustrated in FIG. 6B and the touch panel 13 detects a touch operation on the button K1 or the button K2, the control unit 21 Determines that there is an electronic settlement instruction based on a detection signal from the touch panel 13.

  On the other hand, when the touch operation is performed on the button K3 indicating “cancel” illustrated in FIG. 6B and the touch panel 13 detects the touch operation on the button K3 (S20 “No”), the control unit 21 performs It is determined that there is no electronic settlement instruction based on the detection signal. The control unit 21 ends this processing.

  In addition, when the touch operation is performed on the button K4 indicating the “reminder later” and the touch panel 13 detects the touch operation on the button K4, the control unit 21 determines whether or not the electronic payment instruction is issued based on the detection signal from the touch panel 13. Is temporarily suspended.

  The control unit 21 determines that the touch panel 13 has received a payment instruction for the content of the electronic payment displayed on the display unit 12 from the user (S20 “Yes”: operation step), that is, as illustrated in FIG. When there is a touch operation on the button K1 or the button K2, the communication unit 14 transmits characters related to electronic payment to the payment server 30 (S21: communication step).

  Specifically, when the button K1 indicating “collective payment” shown in FIG. 6B is touch-operated, the communication unit 14 displays the character P10 of the name of the bank (eg, XYZ bank) in which the user has an account, The character P11 of the account holder, the character P13 of the account number, the character T22 indicating the name of the due date and the due date, and the character T23 indicating the name of the total payment and the amount thereof are transmitted to at least the settlement server 30. When the button K2 indicating “installment payment” is touched, the communication unit 14 replaces the character T23 with the name of the minimum installment payment and the character T24 indicating the amount, instead of the character T23. The contents are transmitted to the settlement server 30.

  The payment server 30 performs electronic payment based on the received characters related to electronic payment. For example, the payment server 30 performs electronic payment such that a payment amount designated by the user is withdrawn from a bank in which the user has an account and the withdrawn payment amount is paid to a billing source. Then, the payment server 30 transmits the result of the electronic payment to the communication unit 14 of the information terminal device 10.

  The display control unit 29 of the information terminal device 10 causes the display unit 12 to display the result of the electronic payment received by the communication unit 14 under the control of the control unit 21 (S22). After S22, the control unit 21 ends the present process.

  As described above, in the first embodiment, the identification information extracting unit 24 extracts the identification information indicating the billing source from the bill image acquired by the image acquiring unit 22. The type determination unit 25 determines the type of the bill corresponding to the identification information extracted by the identification information extraction unit 24. The area identification unit 26 reads a format corresponding to the type of invoice determined by the type determination unit 25 from the format storage unit 153 that stores in advance the format of each invoice for each type of invoice, and reads the read invoice. Based on the format, an area where a character image related to electronic payment is included in the bill image is specified. The character recognition unit 27 recognizes a character in the area specified by the area specifying unit 26. The payment information extraction unit 28 extracts characters related to electronic payment from the characters in the area where the character recognition unit 27 has recognized the characters. The display control unit 29 causes the display unit 12 to display characters related to the electronic payment extracted by the payment information extraction unit 28. The touch panel 13 receives a payment instruction from the user regarding the content of the electronic payment displayed on the display unit 12. When the touch panel 13 receives a payment instruction, the communication unit 14 transmits characters related to electronic payment to the payment server 30. For this reason, the payment server 30 can perform electronic payment based on the transmitted characters related to electronic payment. Therefore, electronic settlement of a bill without a barcode can be performed. In addition, since characters related to electronic payment can be generated and transmitted from various invoice images, electronic payment can be performed for various invoices, and convenience can be improved.

  In addition, the user information storage unit 151 stores user authentication information and personal information of the user in advance in association with each other. The user authentication unit 23 determines whether the input information for user authentication input to the touch panel 13 matches the authentication information stored in the user information storage unit 151. The image acquiring unit 22 acquires a bill image when the user authentication unit 23 determines that they match. That is, only when it is confirmed that the user is the user, the bill image for the user is acquired. Therefore, only the user himself / herself can give an instruction for electronic payment, and the security of electronic payment can be ensured.

  Further, the payment information extracting unit 28 extracts at least one of the characters of the total payment amount and the installment payment amount from among the characters in the character recognized by the character recognizing unit 27 as the characters related to the electronic payment, and also stores the user information storage unit. From the personal information of the user stored in 151, at least the character P10 of the name of the bank (for example, XYZ bank) in which the user has an account, and the characters of the account holder and account number are acquired. This eliminates the need for the user to input characters related to electronic payment, that is, characters of the total payment amount and installment payment amount, and characters of the user's bank name, account holder, and account number. Therefore, erroneous input regarding electronic payment can be prevented, and security can be ensured.

  Further, when the user authentication unit 23 determines that there is a match, the image acquisition unit 22 acquires a bill image photographed by the photographing unit 16. That is, only when it is confirmed that the user is the user, the bill image for the user is acquired. Therefore, only the user himself / herself can give an instruction for electronic payment, and the security of electronic payment can be ensured.

  In addition, when the user authentication unit 23 determines that there is a match, the image acquisition unit 22 acquires the bill image received by the communication unit 14. That is, only when it is confirmed that the user is the user, the bill image for the user is acquired. Therefore, only the user himself / herself can give an instruction for electronic payment, and the security of electronic payment can be ensured.

  Further, the identification information extraction unit 24 includes an analysis unit 241 and an identification image area specifying unit 242. The analysis unit 241 analyzes the layout of the bill image acquired by the image acquisition unit 22. The identification image area identification unit 242 identifies an identification image area including an identification image indicating identification information in the bill image that has been subjected to layout analysis by the analysis unit 241. Then, the identification information extraction unit 24 extracts the identification information indicated by the identification image by performing image recognition or character recognition on the identification image region specified by the identification image region specifying unit 242. Therefore, there is no need for the user to specify the location where the identification image exists in the bill image, and the identification information can be accurately extracted from the bill image.

  Next, an information processing system 1A according to a second embodiment will be described with reference to FIG. The information processing system 1A according to the second embodiment is different from the information processing system according to the first embodiment in that the payment server 30A performs a part of the functions performed by the information terminal device 10 according to the first embodiment. It is different from the processing system 1.

  Specifically, instead of the user authentication unit 23, the type determination unit 25, and the area identification unit 26 of the information terminal device 10 according to the first embodiment, the server control unit 311 of the payment server 30A includes a user authentication unit 323, The storage unit of the settlement server 30A includes a type determination unit 325 and an area identification unit 326, and replaces the user information storage unit 151, the identification information storage unit 152, and the format storage unit 153 of the information terminal device 10 according to the first embodiment. 350 includes a user information storage unit 351, an identification information storage unit 352, and a format storage unit 353.

  The information processing system 1A of the second embodiment is a system in which the information terminal device 10A and the settlement server 30A can communicate via a network.

  The settlement server 30A includes a server control unit 311.

  The server control unit 311 includes a processor, a random access memory (RAM), a read only memory (ROM), and the like. The processor is a CPU (Central Processing Unit), an MPU, an ASIC, or the like. The server control unit 311 functions as a user authentication unit 323, a type determination unit 325, and an area identification unit 326 by executing the server control program stored in the ROM or the storage unit 350 by the processor. Each of the above-described components of the server control unit 311 may be configured by a hardware circuit without performing the operation based on the above-described server control program.

  The user authentication unit 323 matches the input information for user authentication input to the touch panel 13 of the information terminal device 10A and transmitted from the information terminal device 10A with the authentication information stored in the user information storage unit 351. It is determined whether or not to perform. If the user authentication unit 323 determines that they match, the result of the authentication match is transmitted from the server communication unit 314 of the payment server 30A to the communication unit 14 of the information terminal device 10A. Then, in the information terminal device 10A, by receiving the result of the authentication matching, the photographing unit 16 can photograph the bill for the user, and the image acquiring unit 22 converts the bill image, which is the image of the bill for the user, into an image. Is obtained.

  The information terminal device 10A transmits to the settlement server 30A the identification information extraction unit 24 that extracts identification information indicating a billing source from the bill image acquired by the image acquisition unit 22 and the identification information extracted by the identification information extraction unit 24. And a communication unit 14.

  The settlement server 30A includes a type determination unit 325 that determines the type of invoice corresponding to the identification information extracted by the identification information extraction unit 24, received from the communication unit 14, and a format of each invoice for each type of invoice. A format storage unit 353 stored in advance and a format corresponding to the bill type determined by the type determination unit 325 are read from the format storage unit 353, and electronic billing is performed on the bill image based on the read bill format. An area specifying unit 326 that specifies an area including a character image, and a server communication unit 314 that transmits the area specified by the area specifying unit 326 to the information terminal device 10A are provided.

  The storage unit 350 of the settlement server 30A includes a user information storage unit 351, an identification information storage unit 352, and a format storage unit 353. The user information storage unit 351 stores in advance authentication information of the user and personal information of the user in association with each other. The identification information storage unit 352 stores in advance identification information indicating a billing source such as a bank or a credit company. The format storage unit 353 stores in advance the format of each bill for each type of bill.

  When the server communication unit 314 receives the identification information extracted by the identification information extraction unit 24 transmitted from the information terminal device 10A, the type determination unit 325 determines the type of the bill corresponding to the received identification information by the identification information. The determination is made using the storage unit 352.

  The area specifying unit 326 reads out a format corresponding to the type of invoice determined by the type determination unit 25 from the format storage unit 353, and based on the format of the read out invoice, a character image related to electronic payment is included in the invoice image. Identify the included area.

  When the area is specified by the area specifying unit 326, information indicating the specified area is transmitted from the server communication unit 314 of the payment server 30A to the communication unit 14 of the information terminal device 10A.

  The information terminal device 10A includes a character recognizing unit 27 for recognizing characters in the area received by the communication unit 14 from the server communication unit 314, a display unit 12, and a character recognizing unit. Payment information extraction unit 28 for extracting characters related to electronic payment, display control unit 29 for displaying characters related to electronic payment extracted by payment information extraction unit 28 on display unit 12, and contents of electronic payment displayed on display unit 12. And a touch panel 13 that accepts a settlement instruction for from the user. When the touch panel 13 receives the payment instruction, the communication unit 14 transmits characters related to the electronic payment to the payment server 30A.

  The payment server 30A performs electronic payment based on characters related to the electronic payment from the communication unit 14 received by the server communication unit 314. For example, the payment server 30A performs an electronic payment in which a payment amount designated by the user is withdrawn from a bank in which the user has an account and the withdrawn payment amount is paid to a billing source. Payment server 30A transmits the result of the electronic payment to communication unit 14 of information terminal device 10A.

  The display control unit 29 of the information terminal device 10A causes the display unit 12 to display the result of the electronic payment received by the communication unit 14 under the control of the control unit 21.

  As described above, in the second embodiment, similarly to the case of the above-described first embodiment, electronic payment of a bill without a barcode can be performed. In addition, since characters related to electronic payment can be generated and transmitted from various invoice images, electronic payment can be performed for various invoices, and convenience can be improved.

  In each of the above embodiments, a bill from the bank that is the billing source (ABC Bank in FIG. 5) to the user is described as an example, but a bill from the credit company to the user, a merchandise store or Electronic payment can also be performed for bills to users from service providers.

1, 1A information processing system 10, 10A information terminal device 11 control unit 12 display unit 13 touch panel (operation unit)
Reference Signs List 15 storage unit 16 imaging unit 21 control unit 22 image acquisition unit 23 user authentication unit 24 identification information extraction unit 25 type determination unit 26 area specification unit 27 character recognition unit 28 payment information extraction unit 29 display control unit 30, 30A payment server 151 user Information storage unit 152 Identification information storage unit 153 Format storage unit 241 Analysis unit 242 Identification image area specification unit 314 Server communication unit 323 User authentication unit 325 Type determination unit 326 Area specification unit 350 Storage unit 351 User information storage unit 352 Identification information storage unit 353 Format storage unit

Claims (8)

An image acquisition unit that acquires an invoice image that is an image of an invoice for the user;
An identification information extraction unit that extracts identification information indicating a billing source from the bill image acquired by the image acquisition unit,
A type determination unit that determines the type of bill corresponding to the identification information extracted by the identification information extraction unit,
A format storage unit for storing in advance the format of each invoice for each type of invoice,
A format corresponding to the type of invoice determined by the type determination unit is read from the format storage unit, and an area including a character image related to electronic settlement is specified in the invoice image based on the format of the read invoice. An area specifying unit to be
A character recognition unit that recognizes a character in the area identified by the area identification unit;
A display unit,
A payment information extraction unit that extracts a character related to electronic payment among the characters in the area in which the character recognition unit has recognized the character;
A display control unit that causes the display unit to display characters related to the electronic payment extracted by the payment information extraction unit;
An operation unit that receives a payment instruction from a user for the content of the electronic payment displayed on the display unit,
A communication unit that, when the operation unit receives the instruction for the payment, transmits characters related to the electronic payment to a payment server. A user information storage unit that stores in advance authentication information of the user and personal information of the user in association with each other;
Input information for user authentication input to the operation unit, a user authentication unit that determines whether the authentication information stored in the user information storage unit matches,
The information processing apparatus according to claim 1, wherein the image acquisition unit acquires the bill image when the user authentication unit determines that the two match.   The payment information extraction unit extracts, as characters relating to the electronic payment, at least one of characters of the total payment amount and the installment payment amount from among the characters in the area where the character recognition unit has recognized the characters, and stores the user information. The information processing apparatus according to claim 2, wherein at least characters of a bank name, an account holder, and an account number where the user has an account are acquired from the personal information of the user stored in the unit. There is a shooting unit that shoots invoices,
4. The information processing apparatus according to claim 2, wherein the image acquisition unit acquires a bill image photographed by the photographing unit when the user authentication unit determines that they match. 5. The communication unit receives the bill image,
4. The information processing apparatus according to claim 2, wherein the image acquisition unit acquires the bill image received by the communication unit when the user authentication unit determines that they match. 5.   The identification information extraction unit specifies an analysis unit that performs layout analysis of the bill image acquired by the image acquisition unit, and identifies an identification image region including an identification image indicating identification information in the bill image that has been layout analyzed by the analysis unit. 6. An identification image area specifying unit that performs identification or character recognition of the identification image area specified by the identification image area specifying unit, thereby extracting identification information indicated by the identification image. An information processing device according to any one of the above. An image acquisition step of acquiring an invoice image that is an image of an invoice for the user;
An identification information extracting step of extracting identification information indicating a billing source from the bill image acquired in the image acquiring step,
A type determination step of determining the type of bill corresponding to the identification information extracted in the identification information extraction step,
From the format storage unit that stores in advance the format of each invoice for each type of invoice, read the format corresponding to the type of invoice determined in the type determination step, based on the format of the read invoice, An area specifying step of specifying an area including a character image related to electronic payment in the bill image,
A character recognition step of recognizing characters in the area identified in the area identification step,
A payment information extraction step of extracting a character related to electronic payment among the characters in the area recognized in the character recognition step;
A display control step of displaying characters related to electronic payment extracted in the payment information extraction step on a display unit;
An operation step of receiving a payment instruction from a user regarding the content of the electronic payment displayed on the display unit,
A communication step of transmitting characters related to the electronic payment to a payment server when the payment instruction is received in the operation step. An information processing system in which the information processing device and the payment server can communicate via a network,
The information processing device,
An image acquisition unit that acquires an invoice image that is an image of an invoice for the user;
An identification information extraction unit that extracts identification information indicating a billing source from the bill image acquired by the image acquisition unit,
A communication unit that transmits the identification information extracted by the identification information extraction unit to the payment server,
The payment server,
A type determination unit that receives the communication unit and determines a type of a bill corresponding to the identification information extracted by the identification information extraction unit;
A format storage unit for storing in advance the format of each invoice for each type of invoice,
A format corresponding to the type of invoice determined by the type determination unit is read from the format storage unit, and an area including a character image related to electronic payment is specified in the invoice image based on the format of the read invoice. An area specifying unit to be
A server communication unit that transmits the area identified by the area identification unit to the information processing apparatus,
The information processing device,
A character recognition unit for recognizing characters in the area received by the communication unit from the server communication unit;
A display unit,
A payment information extraction unit that extracts characters related to electronic payment among the characters in the area where the character recognition unit has recognized the characters
A display control unit that causes the display unit to display characters related to the electronic payment extracted by the payment information extraction unit;
An operation unit configured to receive a payment instruction from a user regarding the content of the electronic payment displayed on the display unit,
The communication unit, when the operation unit receives the instruction of the payment, transmits a character related to the electronic payment to the payment server,
The information processing system, wherein the payment server performs electronic payment based on character information related to the electronic payment from the communication unit received by the server communication unit.

Images