JP2022055011A - Information processing device, information processing method, and program - Google Patents

Abstract

To facilitate determining the size of a photographed document.SOLUTION: A program causes a computer to function as: acquisition unit that obtains a photographic image obtained by photographing a document; first determination means that determines a similarity between the photographic image and each of at least one past document image that is stored in a storage area; and second determination means that determines a value of the document size of a past document image for which the similarity determined by the first determination means is a fixed value or higher, as the document size of the document.SELECTED DRAWING: Figure 5

Description

The present disclosure relates to information processing techniques for determining document size.

Due to the revision of the Electronic Bookkeeping Law in September 2015, smartphones or digital cameras, etc., that meet the specified resolution will be subject to devices that can read and electronically store national tax-related documents such as receipts and invoices. rice field. On the other hand, when a person other than the recipient (that is, the person who is obliged to keep the books) reads the national tax-related documents, it is necessary to save the information on the size according to the provisions of the Electronic Bookkeeping Law.

Patent Document 1 discloses a technique for confirming whether or not the pixel number size of the document area acquired from the captured image is equal to or larger than the standard pixel number size corresponding to the actual size of the document, and notifying the user of the confirmation result. Has been done.

Japanese Unexamined Patent Publication No. 2018-182451

However, Patent Document 1 merely determines whether or not the pixel number size of the document image read by a smartphone, a digital camera, or the like satisfies a predetermined resolution. That is, the actual document size cannot be estimated. For this reason, when a person other than the recipient takes a picture of a document, it is necessary to place a measure or the like that can measure the size next to the national tax-related document and take the picture together with the measure, which is complicated.

Therefore, it is an object of the present disclosure to easily determine the size of the photographed document.

In the program according to one aspect of the present disclosure, a computer obtains a photographed image obtained by photographing a document, the photographed image, and at least one or more past documents stored in a storage area. The document size of the past document image whose similarity is equal to or greater than a certain value is determined as the document size of the document, and the first determining means for determining the similarity between the image and the image. It is characterized by functioning as a second determination means.

According to the present disclosure, the size of the photographed document can be easily determined.

It is an overall block diagram including an information processing apparatus. It is a hardware block diagram of an information processing apparatus. It is a software block diagram of an information processing apparatus. This is the document size registration screen displayed by the mobile application of the information processing device. It is a flowchart processed by a mobile application of an information processing apparatus. This is the document size registration screen displayed by the mobile application of the information processing device. It is a flowchart processed by a mobile application of an information processing apparatus. This is the same document selection screen displayed by the mobile application of the information processing device.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present disclosure, and not all combinations of features described in the present embodiment are essential for the means of solving the present disclosure. The same configuration will be described with the same reference numerals.

Prior to the description of the embodiment, a supplementary explanation regarding the acquisition of the document size will be given. The Electronic Bookkeeping Law, which has been in force since July 1998 (formally, "Law Concerning Special Provisions for Saving National Tax-related Book Documents Created Using Electronic Computers (Act No. 25 of 1998)") , Partially revised in April 2005. After the revision, receipts and invoices of less than 30,000 yen will be subject to electronic storage if the approval of the tax office chief in charge is obtained in advance.

There was a further revision in September 2015, and the requirement to limit the reading device to "integrated with the manuscript stand" was abolished, and smartphones, digital cameras, etc. became available. When reading national tax-related documents, it is necessary to save information on the size (not necessary when the documents are A4 or smaller and the recipient etc. reads them). However, when using a smartphone, a digital camera, or the like, although it is possible to roughly estimate the document size from the camera and various sensors, it is difficult to determine the document size with an accuracy equivalent to that of a scanner.

Therefore, in the following, an embodiment in which the size of the read document is determined with higher accuracy by using the past document image similar to the captured document image will be described.

<Embodiment 1>
As an example of the information processing apparatus according to the present embodiment, a mobile terminal will be described as an example. A mobile terminal is an example of a mobile communication terminal, and is a terminal that is equipped with a wireless communication function and can be used in any place.

FIG. 1 is a diagram showing an example of the appearance of a mobile terminal. FIG. 1A shows the front side portion 101 of the mobile terminal on the front side of the mobile terminal 100. A touch panel 102 is provided on the front surface portion 101 of the mobile terminal. The touch panel 102 is an example of a display unit such as a display, and has two functions of output (display) and input. FIG. 1B shows the back side portion 103 of the mobile terminal on the back side of the mobile terminal 100 of FIG. 1A. The back surface 103 of the mobile terminal includes a camera 104 for capturing an image. FIG. 1C shows a subject 105 for which an image is taken by the camera 104 of the mobile terminal 100. In the present embodiment, the user of the mobile terminal 100 takes a picture of the subject 105 through the mobile application described later, so that the process described later on the mobile terminal 100 is started. The subject 105 described in this embodiment is, for example, a paper document such as a receipt or an invoice. The mobile application described later can capture an image of the subject 105 and output the image to the touch panel 102.

FIG. 2 is a diagram showing an example of the hardware configuration of the mobile terminal 100. The mobile terminal 100 has a CPU 201, a RAM 202, a ROM 203, an Input / Output interface 204, a NIC 205, and a camera unit 206. The CPU (Central Processing Unit) 201 is a unit that executes various programs of a mobile application and realizes various functions. The RAM (Random Access Memory) 202 is a unit that stores various types of information. The RAM 202 is also a unit that is also used as a temporary working storage area for the CPU 201. The ROM (Read Only Memory) 203 is a unit for storing various programs and the like. For example, the CPU 201 loads the program stored in the ROM 203 into the RAM 202 and executes the program. Further, the CPU 201 executes processing based on a program stored in an external storage device such as a flash memory, an HDD (Hard Disk Drive), or an SSD (Solid State Disk). As a result, the software configuration that functions in the mobile terminal 100 as shown in FIG. 3 and the processing of each step of the sequence described later are realized. Note that all or part of the functions of the mobile terminal 100 and the processing related to the sequence described later may be realized by using dedicated hardware.

The Input / Output interface 204 transmits display data to the touch panel 102 and receives input information transmitted from the touch panel 102. The NIC (Network Interface Card) 205 is a unit for connecting a mobile terminal 100 to a network (not shown). The NIC 205 is conceptual and may be wirelessly connected to the network. The camera unit 206 is connected to the camera 104 and captures an image of the subject 105 into the mobile terminal 100. The bus 207 is a data communication path for transmitting and receiving between the above-mentioned units.

FIG. 3 is a diagram showing an example of the software configuration of the mobile terminal 100. Hereinafter, the software configuration of the mobile terminal 100 in the present embodiment will be described. The program that realizes the function (module unit) in each software (application) shown in FIG. 3 is stored in the ROM 203 or the like of the mobile terminal 100.

The user of the mobile terminal 100 can download and install the mobile application (hereinafter referred to as a mobile application) 302 by using the installation function of the OS of the mobile terminal 100. Further, the mobile application 302 can be executed by the CPU 201. The mobile application 302 performs various data processing on the image of the subject 105 captured via the camera unit 206. The mobile application 302 has a plurality of module units. Specifically, the mobile application 302 includes a main control unit 303, an information display unit 304, an operation information acquisition unit 305, an image analysis unit 306, an OCR unit 307, a storage unit 308, a DB unit 309, a coordinate processing unit 310, and an image. It has a feature amount extraction unit 311.

The main control unit 303 controls the mobile application 302 for the mobile terminal 100, and performs instructions, display control, and management for the module units 303 to 311 described later. The information display unit 304 provides the user interface (UI) of the mobile application 302 according to the instruction from the main control unit 303.

FIG. 4 is a diagram showing an example of a screen that provides a UI (UI for a mobile terminal) of the mobile application 302. The mobile terminal screen 400 is displayed on the touch panel 102 of the mobile terminal 100. Further, in the display area 401 on the mobile terminal screen 400, an image captured via the camera 104 is displayed. The operation information acquisition unit 305 accepts the user's operation on the image or the like via the displayed UI. The UI form (position, size, range, arrangement, display content, etc.) of the mobile application 302 is not limited to the form shown in the figure, and any configuration can be adopted.

Returning to FIG. 3 again, each module portion of the mobile application 302 will be described. The operation information acquisition unit 305 acquires user operation information for the UI of the mobile application 302 displayed by the information display unit 304, and notifies the main control unit 303 of the acquired information. For example, when the user touches the display area by hand, the operation information acquisition unit 305 senses the information on the touched position on the screen and transmits the information on the sensed position to the main control unit 303.

The image analysis unit 306 analyzes an image acquired via the camera unit 206, that is, an image taken by the camera 104, and generates layout information. The layout information includes a rectangular area extracted from the image and its type (characters, figures, lines, tables, etc.). Further, the image analysis unit 306 performs paper surface detection processing and distortion correction processing on the camera input image. Further, the camera unit 206 not only can perform still shooting, but also has a moving image shooting function or a continuous shooting function that can acquire still images at high speed.

The OCR unit 307 performs OCR (optical character recognition) processing on the rectangular region extracted by the image analysis unit 306. The storage unit 308 stores the value acquired by the operation information acquisition unit 305, that is, the value input by the user via the UI of the mobile application 302. Further, the storage unit 308 stores data and the like managed by the DB unit 309, which will be described later.

The DB unit 309 has a database function, and manages coordinate information of a rectangular area corresponding to a document taken in the past, a feature amount of an image, an image after image processing, and the like. The coordinate processing unit 310 performs a position specifying process of an image and a coordinate conversion process when registering coordinate information in a database. The coordinate processing unit 310 performs image position specifying processing using the feature amount of the image (hereinafter referred to as the image feature amount) extracted by the image feature amount extraction unit 311 described later.

The image feature amount extraction unit 311 performs an image feature amount extraction process on the image taken by the camera. The image feature amount is information for specifying a characteristic part in an image, and is used, for example, for specifying a corresponding position of a local image in an entire image. Examples of the method for extracting the image feature amount include a determination method such as SIFT, SURF, and HOG. The extracted image features are used to identify the position and to determine the similarity between the images.

FIG. 5 is a processing flow of the mobile application 302. Hereinafter, a process of determining a document size from an image obtained by photographing a subject using the mobile application 302 will be described. As described above, the flowchart shown in FIG. 5 is a process executed by the mobile application 302 of the mobile terminal 100. That is, it is realized by the CPU 201 expanding the program of the mobile application 302 stored in the ROM 203 into the RAM 202 and executing the program.

This flow is started by taking a picture of the subject 105 with the mobile terminal 100 and acquiring the obtained image by the mobile application 302. The symbol "S" in the description of each process means that the step is a step in the flowchart. In S501, the main control unit 303 acquires a photographed image including the subject 105 from the camera unit 206. In S502, the main control unit 303 instructs the image analysis unit 306 to analyze the captured image acquired in S501. The image analysis unit 306 extracts the document portion from the captured image after correcting the distortion. Further, the image analysis unit 306 may perform color correction and binarization processing as necessary. Further, the main control unit 303 inputs the captured image to the image feature amount extraction unit 311, and the image feature amount extraction unit 311 temporarily stores the extracted image feature amount in the RAM 202.

In S503, the main control unit 303 acquires the focus value at the time of shooting from the camera unit 206. The focus value is represented by, for example, 0.0 (during close-up photography) to 1.0 (during infinity photography).

In S504, the main control unit 303 has taken a picture in the past stored in the DB unit 309 and the parameter group (image of the document part, the number of pixels of the document part, and the image feature amount) of the photographed image determined in S501 to S503. The degree of similarity is determined from the parameters of the image. Examples of the method for determining the degree of similarity between images include a method using color histograms, a method of comparing image features by distance, and a method using machine learning. As a specific flow of determining the similarity, a method of obtaining the similarity from the distance of the image feature amount is given as an example. In this case, first, the area of each item such as a title, a date, and an amount is specified from the feature amount including the character recognition result. Next, the distance between the captured image and the image registered in the DB is compared between the areas of the specified items, and the closer the distance is, the higher the similarity is determined. Further, the accuracy may be improved by adopting a plurality of methods.

Further, the accuracy of determining the similarity may be improved by using the number of pixels of the document portion and the focus value acquired in S503. Specifically, when there are two documents of different sizes with similar characteristics, if the similarity is simply determined based on the image, it will be erroneously determined that the similarity is high. In order to prevent such an erroneous determination, a rough document size is estimated from the number of pixels of the document portion and the focus value acquired in S503. By using the estimated document size as an additional parameter for determining the similarity, it is possible to improve the accuracy of determining the similarity of documents having similar characteristics and different sizes. Supplementally, in the similarity determination as described above, the similarity may be determined in consideration of the handwritten characters, or the similarity may be determined in a state where the handwritten characters are excluded. Further, if the document size estimated from the focus value is not used as an additional parameter, the processing of S503 is unnecessary, and in that case, the process proceeds to S504.

In S505, is there a past document image having a similarity of a certain value (predetermined threshold value) or more from the past document images (hereinafter referred to as past document images) whose similarity is determined by the main control unit 303 in S504? judge. If there is a past document image having a degree of similarity equal to or higher than a certain value, the process transitions to S507. On the other hand, if there is no past document image having a degree of similarity of a certain value or more, the transition to S506 occurs. In S506, the main control unit 303 instructs the information display unit 304 to display the document size registration screen of FIG. In the document size registration screen of FIG. 4, the image of the photographed document portion is displayed in the photographed image display area 401, and the document size input space is displayed in the size input area 402. After confirming the document photographed in the photographed image display area 401, the user can input an arbitrary document size in the size input area and press the registration button 403 to determine (register) the document size.

In S507, the main control unit 303 acquires the document size and the past document image corresponding to the past document image having the highest degree of similarity from the DB unit 309. In the present embodiment, the document size of the past document image having the highest similarity is acquired in order to determine the document size, but the present invention is not limited to this, and any past document image having a similarity of a certain value or more can be used as an arbitrary standard. It may be acquired from the DB unit 309. In S508, the main control unit 303 displays a document size corresponding to the past document image having the highest degree of similarity acquired in S507 and a document size registration screen when a similar document exists using the past document image. Instruct 304. The document size registration screen displayed in S508 when similar documents exist is also referred to as a first registration screen. Further, as shown in FIG. 4, the document size registration screen displayed in S506 when similar documents do not exist is referred to as a second registration screen.

FIG. 6 is a document size registration screen displayed based on the size information of the past document image having the highest similarity when there is a past document image having a similarity with the captured image higher than a predetermined threshold value (FIG. 6). First registration screen). In the captured image display area 601 of FIG. 6, the past document image having the highest degree of similarity to the image input in S501 is displayed instead of the image input in S501. Further, in the size input area 602, the document size corresponding to the past document image having the highest degree of similarity acquired in S507 is input in advance. Here, the explanation that the image displayed in the captured image display area 601 is the image of the estimation source of the estimated size input in the size input area 602 is displayed on the screen. At this time, the degree of similarity may be displayed on the screen in order for the user to recognize how similar the captured image and the past document image are. In addition, the user can change the document size entered in advance. The user can determine (register) the document size with the registration button 603. That is, when the user determines that the document size input in advance is correct and presses the registration button 603, the document size of the photographed image is determined by the document size input in advance. When the registration button 603 is pressed after the user has changed the document size, the document size of the photographed image is determined by the changed document size.

When the registration buttons 403 and 603 are pressed, the main control unit 303 in S509 registers the captured image and the size input in the size input areas 402 and 602 in the DB unit 309 in association with each other. At this time, other parameter groups may be registered in the DB unit 309.

By the above processing, the size of the photographed document can be easily determined. Further, as the number of past document images increases in the DB unit 309, the number of similar images for determining the document size increases, so that the accuracy can be increased. In addition, it is possible to eliminate the need to prepare a tool for measuring the length of a measure or the like at the time of shooting.

<Embodiment 2>
In the first embodiment, the document size corresponding to the past document image having the highest degree of similarity is displayed in S508. However, since the similarity determination performed in S504 may have an error, it is not always possible to make an appropriate determination. Therefore, in the second embodiment, an embodiment in which a processing method is added when a plurality of past document images having similar similarities as candidates exist in the mobile terminal 100 and the mobile application 302 shown in the first embodiment will be described.

FIG. 7 is a processing flow of the mobile application 302 when there are a plurality of past document images having similar similarities as candidates. Note that the portion of FIG. 7 having the same step number as that of FIG. 5 is subjected to the same processing as that of the first embodiment. That is, in S501 to S504 of FIG. 7, a document image is acquired by photographing, image recognition processing is performed on the acquired photographed image, and a parameter group necessary for determining the similarity is acquired, as in FIG. 5 described above. Based on the acquired parameter group, the degree of similarity between the document size photographed image and the past document image is determined.

In S701, the main control unit 303 determines whether or not a past document image having a high degree of similarity exists. At this time, if there is a past document image having the highest similarity and a past document image having a similarity close to the past document image having the highest similarity, a plurality of past document images are used as candidates for determining the document size. judge. The similarity close to the past document image having the highest similarity is the similarity within a predetermined range from the highest similarity. The predetermined range can be set as appropriate. In S702, the main control unit 303 acquires the past document image determined as a candidate from the DB unit 309 and the document size corresponding to the past document image. If there are multiple past document images judged as candidates, the corresponding document size and past document image are acquired. In S703, if there are a plurality of past document images determined to be candidates, the transition is made to S704, and if there are not a plurality of past document images, the transition is made to S508.

FIG. 8 is the same document selection screen. In S704, the main control unit 303 instructs the information display unit 304 to display the same document selection screen. On the same document selection screen, a plurality of past document images acquired in S702 are displayed in the same document candidate display area 801 as candidate images for determining the document size, and are in a selectable state. The past document image selected by the user as the same document is regarded as a similar document, and the process proceeds to S508. As shown in FIG. 8, even when a plurality of past document images having a high degree of similarity exist and cannot fit in the same document candidate display area 801, other past document images having a high degree of similarity are displayed when the touch panel 102 is slid. You may do so.

By the above processing, even when there are a plurality of past document images having a high degree of similarity, the size of the photographed document can be easily determined by allowing the user to select a candidate image.

<Other embodiments>
In the above embodiment, the document size determination when a smartphone is used is given as an example, but the present invention is not limited to this, and for example, a photographed image taken by a digital camera may be processed by a PC. Further, the DB unit having the image database function may not be inside the information processing device but may be in the form of an external server such as a cloud server. In the case of this form, there are merits such as reduction of the load on the terminal and eliminating the risk of data loss. In addition, since more forms can be registered, the accuracy of similarity can be improved.

The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiment is supplied to the system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads the program. This is the process to be executed.

Claims (19)

Computer,
An acquisition method for acquiring captured images obtained by photographing documents, and
A first-determining means for determining the degree of similarity between the captured image and at least one or more past document images stored in the storage area.
The second determination means for determining the value of the document size stored in the storage area in association with the past document image having the similarity of a certain value or more determined by the first determination means as the document size of the document. When,
A program characterized by functioning as. The second determining means sets the value of the document size stored in the storage area in association with the past document image having the highest similarity among the past document images having a certain degree of similarity or more. The program according to claim 1, wherein the size of the document is determined. The program according to claim 1 or 2, wherein the second determination means registers the determined document size in the storage area in association with the captured image. The computer
Further functions as a display control means for displaying the value of the document size stored in the storage area in association with the past document image whose similarity is equal to or higher than a certain value determined by the first determination means on the first registration screen. Let me
The second determination means is characterized in that, when a predetermined operation is performed, the value of the document size displayed on the first registration screen is determined as the document size of the document. The program described in any one of Section 3. The display control means is
The fourth aspect of claim 4 is characterized in that, in addition to the document size value, the past document image associated with the document size value and stored in the storage area is displayed on the first registration screen. Program. The second determination means is
The value of the document size displayed on the first registration screen is changed according to the operation from the user, and then, when the predetermined operation is performed, the changed value is determined as the document size of the document. The program according to claim 4 or 5. The display control means is
When there are a plurality of the past document images having the similarity of a certain value or more among the past document images stored in the storage area, the past document image of one of the plurality of past document images is selected. Display the acceptance selection screen and display
The second decision method is
The program according to claim 4 or 5, wherein the value of the document size of the past document image selected on the selection screen is determined as the document size of the document of the photographed image. The display control means is
If there is no past document image whose similarity is above a certain value, a second registration screen that accepts document size input is displayed.
The program according to claim 4 or 5, further comprising determining the document size received on the second registration screen as the document size of the document. The program according to claim 8, wherein the display control means includes and displays the captured image on the second registration screen. The acquisition means according to any one of claims 1 to 9, wherein among the images obtained by photographing the document, an image of a region corresponding to the document portion is acquired as the photographed image. The program described. The first determination means is any of claims 1 to 10, wherein the similarity is determined by a method using a color histogram, a method of comparing image feature quantities by distance, or a method using machine learning. The program described in item 1. The program according to any one of claims 1 to 11, wherein the first determination means determines the similarity by using the size of the captured image obtained from the focus value and the number of pixels. The program according to any one of claims 1 to 12, wherein the storage area is an area inside the computer. The program according to any one of claims 1 to 12, wherein the storage area is an area outside the computer. Computer,
An acquisition method for acquiring captured images obtained by photographing documents, and
A first-determining means for determining the degree of similarity between the captured image and at least one or more past document images stored in the storage area.
A display control means for displaying the value of the document size stored in the storage area in association with the past document image having a similarity of a certain value or more determined by the first determination means on the registration screen.
A program characterized by functioning as. The steps to acquire the captured image obtained by capturing the document, and
A step of determining the degree of similarity between the captured image and at least one or more past document images stored in the storage area, respectively.
A step of determining the value of the document size stored in the storage area in association with the determined past document image having a similarity of a certain value or more as the document size of the document.
A control method characterized by having. An acquisition method for acquiring captured images obtained by photographing documents, and
A first-determining means for determining the degree of similarity between the captured image and at least one or more past document images stored in the storage area.
The second determination means for determining the value of the document size stored in the storage area in association with the past document image having the similarity of a certain value or more determined by the first determination means as the document size of the document. When,
An information processing device characterized by having. The steps to acquire the captured image obtained by capturing the document, and
A step of determining the degree of similarity between the captured image and at least one or more past document images stored in the storage area, respectively.
A step of displaying the value of the document size stored in the storage area in association with the determined past document image having a similarity of a certain value or more on the registration screen.
A control method characterized by having. An acquisition method for acquiring captured images obtained by photographing documents, and
A first-determining means for determining the degree of similarity between the captured image and at least one or more past document images stored in the storage area.
A display control means for displaying the value of the document size stored in the storage area in association with the past document image having a similarity of a certain value or more determined by the first determination means on the registration screen.
An information processing device characterized by having.

Images