JP2023144246A - Information processing device, information processing system, control method, and program - Google Patents

Abstract

To provide a mechanism that can determine whether a captured image is clear or not, depending on the characteristics of the image.SOLUTION: An information processing device according to the present invention acquires an image of a target, and includes determining means that determines a criterion for determining whether the image is clear on the basis of the type of an object, and notification means that notifies whether the image is clear on the basis of the determination result of whether the image is clear for each divided small region and the criterion.SELECTED DRAWING: Figure 1

Description

The present invention relates to techniques for acquiring and processing images.

Although the application process for goods and services has been digitized and can now be applied via network, there are still some documents that must be attached to the application. In order to cope with this, the application source user (applicant or sales representative) uses the camera function of a mobile device or smartphone to take a photograph of the document, and sends (uploads) the document image to the application receiving server via the network. However, a method is adopted in which the server uses OCR (optical character recognition) to recognize necessary items from a document image and register them as digital data.

However, the images taken by the requesting user, who is a general user, are often unclear, and in that case, the server cannot recognize the necessary items, and it is necessary to request the requesting user to retake the image and retransmit it. Such rework is a hassle and burden for both the application source user and the application recipient. Here, if it is known whether or not re-photographing is necessary before uploading the photographed image to the server, there is no rework and efficiency can be improved.

Patent Document 1 discloses a mechanism that detects the outline of a form included in a form image obtained by photographing the form, and notifies the user whether re-photography is necessary depending on whether the detected outline of the form is complete or not. has been done.

JP2010-204806A

However, Patent Document 1 only determines whether the outer shape detected from the form image is complete, but does not mention determining whether the form image is clear.

Further, the characteristics (complexity, etc.) of the captured image vary depending on the structure, contents, etc. of the form. Therefore, the criteria for determining whether a photographed image is clear differs depending on the type of document and the characteristics of the image.

Furthermore, even within a single form image, determination of whether the image is clear or not may differ depending on the partial area due to differences in brightness or the like. In addition, since there are no boundaries (edges) that are subject to sharpness determination for areas with white backgrounds or solid colors, the determination itself cannot be performed, and how much of the entire image can be determined for sharpness depends on the type of document, etc. It depends.

Therefore, it is an object of the present invention to provide a mechanism that can determine whether a photographed image is clear or not.

The present invention provides an information processing device that acquires an image of a target, comprising: a determining unit that determines a criterion for determining whether or not the image is clear based on characteristic information of the image; The present invention is characterized by comprising a notification unit that notifies whether the image is clear or not based on the determination result of whether the image is clear or not for each small area and the criterion.

According to the present invention, it is possible to provide a mechanism that can determine whether a photographed image is clear or not.

A diagram showing an example of the configuration of an information processing system 100 according to an embodiment of the present invention A diagram showing an example of a hardware configuration applicable to the information processing device 101 and the server 102. Flowchart showing an example of overall processing in the information processing system 100 Flowchart showing an example of threshold setting processing Flowchart showing an example of image sharpness determination processing Flowchart showing an example of OCR result reflection processing Flowchart showing an example of OCR result confirmation processing Data configuration diagram showing an example of a threshold setting table Diagram showing an example of block extraction Diagram showing an example of small area division Diagram showing an example of small area division Screen image showing an example of the judgment result OK screen Screen image showing an example of a judgment result NG screen Screen image showing an example of OCR result OK screen Screen image showing an example of an OCR result NG screen

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

As shown in FIG. 1, the information processing system 100 of this embodiment has a configuration in which an information processing device 101 and a server 102 are communicably connected via a network 103.

The information processing apparatus 101 captures an image of a form (hereinafter referred to as a photographed image) using a camera or the like, and performs a sharpness determination. The information processing apparatus 101 acquires a threshold value setting table 800 from the server 102 for determining the threshold value necessary for determining the sharpness of a photographed image.

Note that in this embodiment, the information processing apparatus 101 acquires the threshold setting table 800 managed by the server 102 and determines the sharpness of the captured image; however, the information processing apparatus 101 is not limited to this. may manage the threshold value setting table 800, or the information processing apparatus 101 may provide the necessary data to the server 102, and the server 102 may perform the sharpness determination of the photographed image.

The information processing device 101 is a portable terminal such as a mobile phone, a smartphone, a tablet terminal, a notebook PC, or a PDA terminal, and has a browser or a dedicated application installed thereon. It is possible to communicate with 102.

An example of a hardware configuration applicable to the information processing apparatus 101 and the server 102 shown in FIG. 1 will be described below with reference to FIG. 2.

In FIG. 2, 201 is a CPU that centrally controls each device and controller connected to the system bus 204. In addition, the ROM 203 or external memory 212 contains BIOS (Basic Input/Output System), which is a control program for the CPU 201, operating system programs (hereinafter referred to as OS), and other programs necessary to realize the functions executed by each server or each PC. Various programs and the like are stored.

202 is a RAM, which functions as the main memory, work area, etc. of the CPU 201; The CPU 201 loads programs and the like necessary for execution of processing from the ROM 203 or the external memory 212 into the RAM 202, and executes the loaded programs to realize various operations.

Further, 205 is an input controller that controls input from the touch sensor 210. A video controller 206 controls display on a display device such as the display 211. Although the display 211 is shown in FIG. 2, the display device is not limited to a CRT, but may be another display device such as a liquid crystal display. These are used by the user of the information processing apparatus 101 as needed.

207 is a memory controller that stores boot programs, various applications, font data, user files, editing files, various data, etc. in an external storage device (hard disk (HD)), flexible disk (FD), or PCMCIA card slot. Controls access to external memory 212 such as CompactFlash (registered trademark) memory connected via an adapter.

A communication I/F controller 208 connects and communicates with external devices via the network 103, and executes communication control processing on the network. For example, communication using TCP/IP is possible.

Note that the CPU 201 enables display on the display 211 by, for example, executing processing to develop (rasterize) an outline font in a display information area in the RAM 202. Further, the CPU 201 is capable of accepting a user's touch operation instruction on the touch sensor 210 on the display 211.

Various programs and various tables to be described later for implementing the present invention are recorded in the external memory 212, loaded into the RAM 202 as necessary, and executed by the CPU 201.

A captured image input controller 209 controls input from the camera 215 via the captured image input processing unit 214 . It is possible to recognize and capture a captured image input from the camera 215.

FIG. 3 is a flowchart illustrating an example of overall processing in the information processing system 100.

The overall processing in FIG. 3 will be explained.

In step S101, the information processing device 101 photographs a document and captures the photographed image. Here, the user logs in to the server 102 in advance via a browser or a dedicated application, displays a photographing screen (not shown), and allows the user to photograph the form. At this time, the user may be allowed to select the size of the form and the type of form (target type).

In step S102, the information processing device 101 performs image adjustment and correction of the captured image. Specifically, in order to make the subsequent image processing more efficient, the color photographed image is converted to grayscale, and keystone correction and the like are performed on the distortion. The threshold setting table 800 sent in is acquired.

In step S105, the information processing apparatus 101 executes threshold value determination processing and determines a threshold value. The threshold value determination process will be explained in detail using FIG. 4.

FIG. 4 is a flowchart illustrating an example of threshold value determination processing.

In step S201, the information processing apparatus 101 determines whether the target type of the photographed image can be specified. For example, it is determined that identification is possible when a user selects a form type from a shooting screen (not shown) before or after shooting, or when identification information indicating the form type is included in a barcode, etc. included in the captured image. do. Further, the form type may be acquired from another information processing system, or the form type may be determined for each function executed by the user. If it can be specified, the process advances to step S202, and if it cannot be specified, the process advances to step S205.

In step S202, the information processing apparatus 101 identifies the target type of the photographed image. For example, if a selection of a form type is accepted from the user, the selected form type is selected, and if identification information indicating the form type is included in a barcode or the like included in the photographed image, the form type indicated by the identification information is selected. Specify as the target type of the captured image.

In step S203, the information processing apparatus 101 compares the specified target type with the conditions of the threshold setting table 800. Specifically, it is checked whether the specified target type exists in the target type 801 of the threshold setting table 800. If it does not exist, the process starts from step S205, as in the case where the target type cannot be specified. The threshold setting table 800 will be described later.

In step S204, the information processing apparatus 101 determines the threshold value 806 of the record matched in the threshold value setting table 800 as the threshold value, and ends this processing flow.

In step S205. The information processing apparatus 101 executes image analysis of the photographed image and acquires the feature amount of the image in step S206.

Here, a specific example of image analysis will be described using FIG. 9.

A form image 900 is an image diagram showing an example of a photographed image targeted this time. The form image 900 includes various areas such as a single-line text area 901, a multi-line text area 902, a table area 903, a ruled text area 904, and a graphic area 905. In the image analysis of this example, each region is extracted as a block group using block selection (for block selection, see Japanese Patent Application Laid-Open No. 2006-195887, etc.), and for text regions, each line of blocks (911 , 912, 914), and the table area is extracted by disassembling the cell blocks (913). Obtain the number of characters for a block containing characters, and obtain the number of ruled lines from the table area or text area surrounding ruled lines. Furthermore, the pixel ratio (ratio of non-white pixels) of the entire image is also obtained.

Note that the image analysis method described above is just an example, and various information (feature amounts) indicating the characteristics of the captured image may be acquired using known techniques or the like.

Returning to the explanation of FIG. 4.

In step S207, the information processing apparatus 101 compares the feature amount acquired in step S206 with the conditions of the threshold setting table 800. Specifically, it is checked whether the acquired feature quantities such as the number of blocks, the number of characters, the pixel rate, the number of ruled lines, etc. conform to each item (802 to 805) in the condition section of the threshold setting table 800.

In step S208, the information processing apparatus 101 determines the threshold value 806 of the record matched in the threshold value setting table 800 as the threshold value, and ends this processing flow.

Note that in this embodiment, the threshold value is determined by checking against a threshold setting table having condition items and determination items, but it is also possible to define a function using each feature amount as a variable and determine the threshold value using the function. Furthermore, a table may be used to determine weighting coefficients for each feature in the function.

This concludes the description of the threshold value setting process in FIG. 4. Here, the threshold value setting table will be described using FIG. 8.

The threshold value setting table 800 is a table for determining a threshold value based on the object type or the feature amount obtained by image analysis. In this example, there are a target type 801 as a condition item related to the target type, a number of blocks 802, a number of characters 803, a pixel rate 804, and a number of ruled lines 805 as condition items related to feature quantities, and each item is defined as a value or a range. If the condition is set and the condition is free, set "*}.In this example, the condition item related to the target type and the condition item related to the feature amount are set separately, but a record of a condition that combines both. In addition, a threshold value 806 is provided as a determination item, but this value indicates the number (absolute value) or ratio (relative value) of small areas, which will be described later. In order to cope with the case where the condition does not match, a default record may be set in which all condition items are set to "*".

Note that in this example, the above condition items are incorporated as feature amounts (feature information) that affect the sharpness determination of a captured image, but the present invention is not limited to this, and the characteristics (especially complexity) of the captured image are What is necessary is to adopt an index that indicates the sharpness of the photographed image and that influences the sharpness determination of the photographed image.

Returning to the explanation of FIG. 3.

In step S106, the information processing device 101 divides the photographed image into small regions.

Here, division of a photographed image into small regions will be explained using FIGS. 10 and 11.

In this embodiment, two types of division methods are assumed. FIG. 10 shows a method of dividing with a fixed number of divisions, and FIG. 11 shows a method of dividing with a fixed division size.

In the method of dividing with a fixed number of divisions in FIG. 10, regardless of the size of the photographed image, the photographed image is equally divided vertically and horizontally by a fixed number into small regions. Even if the size of the photographed image changes, the number of small regions generated by division remains the same, so a threshold value indicating the number can be used. Thresholds indicating ratios can also be used.

In the method of dividing with a fixed division size in FIG. 11, the number of vertical and horizontal divisions of the photographed image differs depending on the size of the photographed image. Since the number of small regions generated by dividing differs depending on the size of the photographed image, a threshold value indicating the number cannot be used, but a threshold value indicating the ratio is used.

Returning to the explanation of FIG. 3.

In steps S107 to S109, the information processing apparatus 101 repeats the process for each divided small area.

In step S108, the information processing apparatus 101 performs sharpness/unsharpness determination on the small area to be processed. In this process, it is determined whether the small region to be processed is clear or unclear using known techniques such as image spectrum analysis and automatic determination using machine learning. By performing sharpness/unsharpness judgment on the divided small areas, even if there is a difference in brightness or darkness within a single captured image, the judgment is made based on the judgment criteria for each small area, so the sharpness/unsharpness can be determined more appropriately and efficiently. Judgments can be made.

In step S110, the information processing apparatus 101 executes image sharpness determination processing. The image sharpness determination process will be explained in detail using FIG. 5.

FIG. 5 is a flowchart illustrating an example of image sharpness determination processing.

In step S301, the information processing apparatus 101 totals the sharpness/unsharpness determination results for each small area. Specifically, the number of small areas judged to be clear and the number of small areas judged to be unclear are totaled, and the number of small areas judged to be clear is divided by the total number of small areas to determine the number of small areas judged to be clear. Find the ratio of small areas.

In step S302, the information processing apparatus 101 determines whether the total result is greater than or equal to the threshold set in step S105. In other words, if the set threshold is a numerical threshold, it is determined whether the number of small areas judged to be sharp is greater than that number, and if it is a threshold set to a ratio, it is determined whether the proportion of small areas judged to be clear is greater than or equal to the set threshold. to decide. If it is determined that it is equal to or greater than the threshold value, the process advances to step S303, and if it is determined that it is not (less than the threshold value), the process advances to step S308.

In step S303, the information processing apparatus 101 displays a determination result OK screen 1200, an example of which is shown in FIG. The determination result OK screen 1200 identifies and displays the sharpness/unsharpness determination result for each small area on the photographed image as a preview of the determination result (1201), and displays the sharpness/unsharpness determination result together with its basis in text (1202). Further, instructions from the user are accepted using an image transmission button 1203 and a reshoot button 1204.

In step S304, the information processing apparatus 101 determines whether an image transmission instruction has been received using the image transmission button 1203. If it is accepted, the process advances to step S305; otherwise (if a reshooting instruction is accepted), the process advances to step S306.

In step S305, the information processing apparatus 101 determines that the sharpness determination based on the threshold value matches the user's determination, sets the subsequent process to “image transmission”, and ends this processing flow.

In step S306, the information processing apparatus 101 transmits a request to update the threshold value setting table 800 to the server 102 because the sharpness determination based on the threshold value and the user's determination do not match. Specifically, it is requested to be stricter, that is, to increase the threshold value beyond the current aggregation result. Upon receiving the update request, the server 102 updates the threshold setting table 800 in step S313.

In step S307, the information processing apparatus 101 performs subsequent processing as "re-imaging" and ends this processing flow.

In step S308, the information processing apparatus 101 displays a determination result NG screen 1300, an example of which is shown in FIG. The determination result NG screen 1300 identifies and displays the sharpness/unsharpness determination result for each small area on the photographed image as a determination result preview (1301), and displays the sharpness/unsharpness determination result together with its basis in text (1302). Further, instructions from the user are accepted using a reshoot button 1303 and an image transmission button 1304.

In step S309, the information processing apparatus 101 determines whether a reshooting instruction has been received using the reshooting button 1303. If accepted, the process advances to step S310; otherwise (if the image transmission instruction is accepted), the process advances to step S311.

In step S310, the information processing apparatus 101 determines that the unsharpness determination based on the threshold value matches the user's determination, sets the subsequent process to “re-imaging”, and ends this processing flow.

In step S<b>311 , the information processing apparatus 101 transmits a request to update the threshold setting table 800 to the server 102 because the blurring determination based on the threshold value and the user's determination do not match. Specifically, it requests that the threshold value be reduced to less than the current aggregation result. Upon receiving the update request, the server 102 updates the threshold setting table 800 in step S313.

In step S312, the information processing apparatus 101 sets the subsequent process to "image transmission" and ends this process flow.

This concludes the explanation of the image sharpness determination process in FIG. 5.

Returning to the explanation of FIG. 3.

In step S112, the information processing apparatus 101 determines whether the subsequent process is "reshooting", and if so, returns to step S101; otherwise (if "image transmission"), proceeds to step S113.

In step S113, the information processing device 101 transmits the captured image to the server 102.

In step S114, the server 102 receives the captured image.

In step S115, the server 102 executes OCR processing on the photographed image, recognizes and acquires acquisition items from the photographed image. Regarding the acquisition items, positions within the form image, keywords serving as landmarks, etc. are set in advance, and acquired by a predetermined processing procedure.

In step S116, the server 102 executes OCR result reflection processing. The OCR result reflection process will be explained in detail using FIG. 6.

FIG. 6 is a flowchart illustrating an example of OCR result reflection processing.

In step S401, the server 102 determines whether the OCR result is OK. Specifically, if all of the required items among the acquisition items are recognized and acquired, it is determined to be OK, and if even one of the required items is not recognized or acquired, it is determined to be NG. For essential items, categories are set in advance for the acquired items. If OK, proceed to step S402; if NG, proceed to step S403.

In step S402, the server 102 determines that the information processing apparatus 101 made the correct decision to send the image, sets the OCR result to OK, and ends this processing flow.

In step S403, the server 102 determines that the information processing apparatus 101 made an incorrect determination to send the image, and updates the threshold setting table 800. Specifically, the threshold value is updated to be stricter, that is, to increase the threshold value beyond the current tally result in the information processing apparatus 101. In step S311, even if the user decides to "transmit an image" and the threshold value is temporarily relaxed, if the OCR result is NG, the threshold value will be returned to the stricter one.

In step S404, the server 102 sets the OCR result to NG and ends this processing flow.

This concludes the explanation of the OCR result reflection process shown in FIG.

Returning to the explanation of FIG. 3.

In step S117, the server 102 transmits the OCR result set in step S116 to the information processing apparatus 101.

In step S118, the information processing apparatus 101 receives the OCR result.

In step S119, the information processing apparatus 101 executes OCR result confirmation processing. The OCR result confirmation process will be explained in detail using FIG. 7.

FIG. 7 is a flowchart illustrating an example of OCR result confirmation processing.

In step S501, the information processing apparatus 101 determines whether the OCR result is OK. If OK, the process proceeds to step S502; if NG, the process proceeds to step S506.

In step S502, the information processing apparatus 101 displays an OCR result OK screen 1400, an example of which is shown in FIG. The OCR result OK screen 1400 displays the transmitted image image as a transmitted image preview (1401), and displays the OCR result in text along with its basis (1402). Further, instructions from the user are accepted using the separate image capture button 1403 and end button 1404.

In step S<b>503 , the information processing apparatus 101 determines whether a separate image shooting instruction has been received using the separate image shooting button 1403 . If it is accepted, the subsequent process is set to "separate image shooting" in step S504, and if not (if the termination instruction is accepted), the subsequent process is set to "end" in step S505, and this processing flow ends.

In step S506, the information processing apparatus 101 displays an OCR result NG screen 1500, an example of which is shown in FIG. The OCR result NG screen 1500 displays the transmitted image as a transmitted image preview (1501), and displays the OCR result in text along with its basis (1502). Further, instructions from the user are accepted using a reshoot button 1503 and an end button 1504.

In step S507, the information processing apparatus 101 determines whether a reshooting instruction has been received using the reshooting button 1503. If it is accepted, the subsequent process is set to "reshoot" in step S508, and if not (if the end instruction is accepted), the subsequent process is set to "end" in step S509, and this process flow ends.

This concludes the explanation of the OCR result confirmation process shown in FIG.

Returning to the explanation of FIG. 3.

In step S120, the information processing apparatus 101 determines whether the subsequent process is "separate image capture" or "re-capture", and if yes, returns to step S101; otherwise, ends this processing flow.

This concludes the explanation of the overall process shown in FIG.

As described above, it becomes possible to appropriately determine whether the image is clear or not, depending on the characteristics of the photographed image, even if there is a difference in brightness or darkness within the image.

Although several embodiments have been described above, the present invention can be implemented as, for example, a system, an apparatus, a method, a computer program, a recording medium, etc. The present invention may be applied to a system consisting of , or may be applied to a device consisting of a single device.

Further, the computer program in the present invention is a computer program that allows a computer to execute the processing method of each flowchart, and the storage medium of the present invention stores a computer program that allows a computer to execute the processing method of each flowchart. Note that the computer program in the present invention may be a computer program for each processing method of each device in each flowchart.

As described above, a recording medium recording a program that implements the functions of the embodiments described above is supplied to a system or device, and the computer (or CPU or MPU) of the system or device reads the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by reading and executing.

In this case, the program itself read from the recording medium will realize the novel function of the present invention, and the recording medium on which the program is recorded constitutes the present invention.

Examples of recording media for supplying programs include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EPROMs, and silicon A disk or the like can be used.

In addition, by executing a program read by a computer, not only the functions of the above-described embodiments are realized, but also the OS (operating system) etc. running on the computer are realized based on the instructions of the program. It goes without saying that this also includes a case where part or all of the processing is performed and the functions of the embodiments described above are realized by the processing.

Furthermore, after the program read from the recording medium is written into the memory of the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board It goes without saying that this also includes a case where a CPU or the like provided in a function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

Moreover, the present invention may be applied to a system made up of a plurality of devices, or to a device made up of one device. It goes without saying that the present invention can also be applied to cases where the present invention is achieved by supplying a program to a system or device. In this case, by reading a recording medium storing a program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

The program may be in the form of an object code, a program code executed by an interpreter, script data supplied to an OS (operating system), or the like.

Further, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. Note that all configurations that are combinations of the above-described embodiments and their modifications are also included in the present invention.

100 Information processing system 101 Information processing device 102 Server 103 Network

The present invention provides an information processing device that acquires an image of a document , including a determining unit that determines a criterion for determining whether or not the image of the document is clear based on the type of the document ; The present invention is characterized by comprising a notification means for notifying whether the image of the document is clear or not based on the determination result of whether the image of the document is clear or not for each small area and the determined criterion.

Claims (8)

An information processing device that acquires an image of a target,
determining means for determining a criterion for determining whether or not the image is clear based on the type of the object;
Notifying means for notifying whether the image is clear or not based on the determination result of whether the image is clear for each divided small region and the criterion;
An information processing device comprising: reception means for receiving instructions regarding the content of the notification by the notification means;
update control means for controlling the determined criteria to be updated based on the notification content by the notification unit and the instruction content received by the reception unit;
The information processing device according to claim 1, further comprising: The information processing apparatus according to claim 1, wherein the reference is a condition value related to the number or ratio of small areas that are determined to be clear. 4. The information processing apparatus according to claim 1, wherein the image is divided into small areas with a fixed number or a fixed size. An information processing device that acquires an image of a target,
determining means for determining a criterion for determining whether or not the image is clear, based on the feature amount acquired from the image;
Notifying means for notifying whether the image is clear or not based on the determination result of whether the image is clear for each divided small region and the criterion;
An information processing device comprising: An information processing system in which an information processing device that acquires a target image and a server that processes the image are communicably connected,
determining means for determining a criterion for determining whether or not the image is clear based on the type of the object;
Notifying means for notifying whether the image is clear or not based on the determination result of whether the image is clear for each divided small region and the criterion;
An information processing system comprising: A method for controlling an information processing device that acquires an image of a target, the method comprising:
a determining step in which the determining means determines a criterion for determining whether or not the image is clear based on the type of the object;
a notification step in which the notification means notifies whether the image is clear or not based on the determination result of whether the image is clear for each divided small region and the criterion;
A method for controlling an information processing device, comprising: A program executable by an information processing device that acquires a target image,
The information processing device,
determining means for determining a criterion for determining whether or not the image is clear based on the type of the object;
Notifying means for notifying whether the image is clear or not based on the determination result of whether the image is clear for each divided small region and the criterion;
A program to function as

Images