JP2021086188A - Image processing system, image processing method, and program - Google Patents

Abstract

To reduce the possibility of false determination for a character string alteration in image data.SOLUTION: An image processing system includes: generation means for generating a learning model by performing machine learning processing based on an altered character image, a character image before an alteration, and an image representing the difference between the altered character image and the character image before the alteration; input means for inputting image data; and estimation means for estimating whether the image data input by the input means includes an altered character, by using the learning model generated by the generation means.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to image processing systems, image processing methods, and programs.

Conventionally, a technique for detecting falsification in image data has been known.

According to Patent Document 1, if the image data is divided into a plurality of groups having pixels having similar brightness values and the result of executing the character recognition process for each group is different from the result of executing the character recognition process without dividing the image data into groups. An image processing device that determines that the image has been tampered with is disclosed.

Japanese Unexamined Patent Publication No. 2009-200794

In Patent Document 1, since the presence or absence of tampering is determined only by the brightness, there is a possibility that it is erroneously determined that the character has been tampered with even if it has not been tampered with because the brightness of the character changes due to ink fading or writing pressure. There is.

The present invention has been made in view of the above problems, and an object of the present invention is to reduce the possibility of erroneous determination of falsification of a character string in image data.

The image processing system of the present invention executes machine learning processing based on a falsified image, an image before falsification, and an image showing a difference between the falsified image and the image before falsification. And the generation means to generate the learning model by
Input means for inputting image data and
It is characterized by having an estimation means for estimating whether or not the image data input by the input means includes a falsified image by using the learning model generated by the generation means.

According to the present invention, it is possible to reduce the probability that an erroneous determination will occur when detecting falsification of a character string in image data.

The schematic diagram which shows an example of the structure of the image processing system which concerns on one Embodiment. (A) A block diagram showing an example of the configuration of an image processing device. (B) A block diagram showing an example of the configuration of the learning device. (C) A block diagram showing an example of a configuration of a tampering detection server. (A) Sequence diagram showing an example of a schematic flow of processing in the learning stage in the image processing system according to one embodiment; (B) Schematic flow of processing in the tampering detection stage in the image processing system according to one embodiment. A sequence diagram showing an example. The schematic diagram which shows an example of the blank learning manuscript. The schematic diagram which shows an example of GUI for accepting the instruction of manuscript reading. (A) Schematic diagram showing an example of a learning original image; (B) Schematic diagram showing an example of a learning falsified image; (C) An example of learning data generated from a learning original image and a learning falsified image Schematic diagram shown. (A) Schematic diagram showing an example of the image to be processed; (B) Schematic diagram showing an example of a bitmap of the falsification detection result; (C) Schematic diagram showing an example of an emphasized image emphasizing the falsified portion. A flowchart showing an example of a specific flow of processing executed by an image processing apparatus in a tampering detection stage. A flowchart showing an example of a specific flow of processing executed by the tampering detection server in the tampering detection stage. The schematic diagram which shows an example of GUI for setting and instruction of tampering detection. (A) Schematic diagram showing an example of GUI for displaying a list of tampering detection results; (B) Schematic diagram showing an example of GUI for displaying details of tampering detection results. The schematic diagram which shows an example of GUI which makes it possible to modify a tampering detection result by a user. (A) Schematic diagram showing a first example of a GUI in which an emphasized image and a comparison image are arranged in contrast; (B) A schematic diagram showing a second example of a GUI in which an emphasized image and a comparison image are arranged in contrast. Figure. (A) Schematic diagram showing an example of a comparison image in the first comparison mode; (B) Schematic diagram showing an example of a comparison image in the second comparison mode; (C) Example of a comparison image in the third comparison mode Schematic diagram shown; (D) Schematic diagram showing an example of a comparative image in the fourth comparison mode. The schematic diagram which shows the other example of GUI which makes it possible to modify a tampering detection result by a user. A flowchart showing an example of a specific flow of display control processing executed by an image processing device when falsification of a target document is detected. The schematic diagram which shows an example of GUI which arranged the emphasized image and the comparative image in contrast in one modification.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate explanations are omitted.

<< 1. System overview >>
FIG. 1 is a schematic view showing an example of the configuration of the image processing system 100 according to the embodiment. The image processing system 100 includes an image processing device 101, a learning device 102, a tampering detection server 103, and an OCR server 104. The image processing device 101, the learning device 102, the tampering detection server 103, and the OCR server 104 are connected to each other via the network 105.

The image processing device 101 may be, for example, a multifunction device (Multi-Function Peripheral (MFP)) having both printing and image reading functions, or may be a digital scanner that exclusively performs image reading. The image processing device 101 includes a reading unit 111 and a display control unit 112. The scanning unit 111 reads the document 11 and generates a scanned image. That is, the scanning unit 111 acquires the scanned image of the document 11. Typically, the manuscript 11 contains a character string, and thus the scanned image contains a character image.

As an example, the image processing device 101 can support the generation of learning data at the learning stage. Specifically, first, the operator writes characters by hand on a blank learning manuscript prepared in advance, and sets the completed learning manuscript in the image processing device 101. The learning manuscript may be, for example, a form-type document having entry fields at one or more known positions. The learning manuscript may have visual identification information (for example, a printed number, a bar code or a two-dimensional code) for uniquely identifying each learning manuscript. The image processing device 101 may be able to print a blank learning document. The scanning unit 111 reads the set learning document and generates the scanned image 12. The scanned image 12 is treated as an original image of the learning manuscript, and is also referred to as a learning original image in the present specification. In addition, the operator or another person falsifies the completed learning manuscript (that is, the original) (for example, by adding strokes with a pen), and the falsified learning manuscript is sent to the image processing device 101. set. The reading unit 111 reads the falsified learning document and generates the scanned image 13. In the present specification, the read image 13 is also referred to as a falsified image for learning. Through repeated writing of characters on the learning manuscript, reading of the original by the image processing device 101, intentional falsification of the learning manuscript, and reading of the falsified version by the image processing device 101, such a learning original image 12 and A plurality of pairs of falsified images 13 for learning are generated. The image processing device 101 transmits these pairs of the learning original image 12 and the learning falsified image 13 to the learning device 102 via the network 105. The learning device 102 executes machine learning using the learning data generated from these pairs, as will be described later. Notwithstanding the above description, the learning original image 12 and the learning falsified image 13 may be generated by a device other than the image processing device 101.

In addition, the image processing device 101 reads a target document that may include handwritten characters and generates a scanned image 21 at the tampering detection stage. In the present specification, the scanned image 21 is also referred to as a processing target image. The image processing device 101 transmits the generated image 21 to be processed to the tampering detection server 103 via the network 105. The display control unit 112 of the image processing device 101 receives the detection result data 32 indicating the result of the tampering detection performed using the processing target image 21 from the tampering detection server 103. Then, the display control unit 112 controls the display of the tampering detection result on the screen based on the detection result data 32. Various examples of display control here will be specifically described later.

The learning device 102 may be an information processing device such as a computer or workstation that executes supervised learning processing. The learning device 102 includes a data processing unit 121, a learning unit 122, and a storage unit 123. The data processing unit 121 stores, for example, a pair of the above-mentioned original learning image 12 and the falsified learning image 13 generated in the image processing device 101 (or another device) in the storage unit 123, and learns from the accumulated pair. Generate data for. The learning unit 122 detects tampering through machine learning processing using learning data generated from a learning image (that is, a pair of a learning original image 12 and a learning tampered image 13) which is a scanned image of a learning manuscript. Generates / updates the trained model (learning model) 41 for. The learning unit 122 stores the generated / updated learned model 41 in the storage unit 123. For example, when a neural network model is used as a machine learning model, the trained model 41 is a data set including parameters such as weights and biases for each node of the neural network. As a machine learning method for generating / updating a neural network model, for example, deep learning using a multi-layer neural network may be used. Some examples of generating training data and generating / updating a trained model in the learning device 102 will be specifically described later. The learning unit 122 provides the learned model 41 to the tampering detection server 103 in response to a request from the tampering detection server 103, which will be described later.

The falsification detection server 103 may be an information processing device such as a computer or a workstation that detects a falsified portion included in the target manuscript using the processing target image 21 received from the image processing device 101. The tampering detection server 103 includes an image acquisition unit 131 and a detection unit 132. The image acquisition unit 131 acquires a processing target image 21 which is a scanned image of the target original. The detection unit 132 detects the falsified portion included in the target manuscript by using the processing target image 21. In the present embodiment, the detection unit 132 uses the above-mentioned learned model 41 provided by the learning device 102 for tampering detection. In particular, in the present embodiment, the detection unit 132 estimates whether or not each of the plurality of pixels in the processed image 21 belongs to the falsification portion by using the trained model 41 (that is, falsification detection for each pixel). .. However, a modified example (that is, falsification detection for each character) in which the detection unit 132 determines whether or not each of the one or more characters in the image 21 to be processed includes a falsified portion will be described later. The detection unit 132 generates detection result data 32 indicating which part of the image 21 to be processed is determined to be tampered with as a result of tampering detection, and provides the generated detection result data 32 to the image processing device 101. To do. For example, the detection result data 32 may include a bitmap indicating whether or not each pixel of the image to be processed 21 belongs to the falsified portion. The falsification detection result indicated by the detection result data 32 is presented to the user by the image processing device 101, and the validity is verified by the user. In order to assist the user in verifying the validity, the image processing device 101 or the falsification detection server 103 generates an enhanced image that emphasizes the pixels determined to belong to the falsified portion in the image 21 to be processed. In the case where the tampering detection server 103 generates the enhanced image, the tampering detection server 103 transmits the generated enhanced image together with the bitmap to the image processing device 101.

In the present embodiment, the application of the processed image 21 to the trained model 41 can be performed on a character-by-character basis. Therefore, the tampering detection server 103 transmits the processing target image 21 to the OCR server 104, and requests the OCR server 104 to recognize the characters included in the processing target image 21. The OCR server 104 may be an information processing device such as a computer or workstation that executes optical character recognition (OCR) in response to a request from the manipulation detection server 103. The OCR server 104 includes a character recognition unit 141. The character recognition unit 141 executes OCR on the processing target image 21 by a known method to recognize the character and the position of the character region in the processing target image 21. Then, the character recognition unit 141 transmits the recognition result data 31 indicating the recognition result to the tampering detection server 103.

<< 2. Device configuration >>
FIG. 2A is a block diagram showing an example of the configuration of the image processing device 101. FIG. 2B is a block diagram showing an example of the configuration of the learning device 102. FIG. 2C is a block diagram showing an example of the configuration of the tampering detection server 103.

(1) Image processing device The image processing device 101 includes a CPU 201, a ROM 202, a RAM 204, a printer device 205, a scanner device 206, a transport device 207, a storage 208, an input device 209, a display device 210, and an external I / F 211. The data bus 203 is a communication line that connects these devices in the image processing device 101 to each other.

The CPU (Central Processing Unit) 201 is a controller for controlling the entire image processing device 101. The CPU 201 executes a boot program stored in advance in the ROM 202, which is a non-volatile memory, to start the OS (Operating System) of the image processing device 101. The CPU 201 executes the controller program stored in the storage 208 on this OS. The controller program is a program for controlling each of the devices of the image processing device 101. The RAM 204 is used as the main storage device of the CPU 201 and provides the CPU 201 with a temporary storage area (ie, a work area).

The printer device 205 is a device that prints an image on paper (also referred to as a recording material or a sheet). The printer device 205 employs any method as a printing method, such as an electrophotographic method using a photoconductor drum or a photoconductor belt, or an inkjet method in which ink is ejected from a minute nozzle array to print an image directly on paper. May be good. The scanner device 206 includes an optical reading device such as a CCD (Charge Coupled Device) that optically scans a document, and converts an electric signal from the optical reading device into image data of a scanned image. The transport device 207 may include a so-called ADF (Auto Document Feeder), and transports the documents set in the ADF one by one to the scanner device 206. The scanner device 206 may be capable of reading the document placed on the platen (not shown) of the image processing device 101 in addition to reading the document conveyed from the transfer device 207.

The storage 208 may be a write / read auxiliary storage device including a non-volatile memory such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage 208 stores various data such as the controller program, setting data, and image data described above. The input device 209 includes, for example, a touch panel or a hard key, and accepts user input such as an operation instruction or information input from the user. The input device 209 transmits an input signal representing the contents of the received user input to the CPU 201. The display device 210 includes, for example, an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube), and displays an image (for example, a user interface image) generated by the CPU 201 on the screen. For example, the CPU 201 determines what kind of operation has been performed by the user based on the pointing position indicated by the input signal received from the input device 209 and the arrangement of the user interface displayed by the display device 210. You may. The CPU 201 controls the operation of the corresponding device or changes the content displayed by the display device 210 according to the determination result.

The external interface (I / F) 211 transmits and receives various data including image data to and from the external device via the network 105. The network 105 may be any type of network, for example a LAN (Local Area Network), a telephone line or a proximity radio (eg, infrared) network. The external I / F211 can receive PDL (Page Description Language) data describing drawing contents for printing from, for example, an external device such as a learning device 102 or a PC (not shown). The CPU 201 interprets the PDL data received by the external I / F 211 to generate image data. The image data may be transmitted to the printer device 205 for printing or to storage 208 for storage. Further, the external I / F 211 can transmit the image data of the scanned image acquired by the scanner device 206 to the tampering detection server 103 for tampering detection, and receive the detection result data from the tampering detection server 103.

(2) Learning device The learning device 102 includes a CPU 231, a ROM 232, a RAM 234, a storage 235, an input device 236, a display device 237, an external I / F 238, and a GPU 239. The data bus 233 is a communication line that connects these devices in the learning device 102 to each other.

The CPU 231 is a controller for controlling the entire learning device 102. The CPU 231 executes a boot program stored in advance in the ROM 232 which is a non-volatile memory to start the OS of the learning device 102. The CPU 231 executes a learning data generation program and a learning program stored in the storage 235 on this OS. The learning data generation program is a program for generating learning data from a pair of a learning original image 12 and a learning falsified image 13. The learning program is a program for generating / updating a trained model (for example, a neural network model) for tampering detection through machine learning. The RAM 234 is used as the main storage device of the CPU 231 and provides the CPU 231 with a temporary storage area (ie, a work area).

The storage 235 may be a writable and readable auxiliary storage device, including a non-volatile memory, such as an HDD or SSD. The storage 235 stores various data such as the above-mentioned program, learning image, learning data, and model data. The input device 236 includes, for example, a mouse and a keyboard, and accepts user input such as an operation instruction or information input from the user. The display device 237 includes, for example, an LCD or a CRT, and displays an image generated by the CPU 231 on the screen. The external I / F 238 transmits / receives data related to the learning process to / from the external device via the network 105. The external I / F 238 may receive, for example, a pair of the learning original image 12 and the learning falsified image 13 from the image processing device 101. Further, the external I / F 238 may transmit the learned model 41 generated / updated through opportunity learning to the alteration detection server 103. The GPU 239 is a processor having a high degree of parallel processing capability, and promotes learning processing for generating / updating a trained model in cooperation with CPU 231.

(3) Manipulation detection server The manipulation detection server 103 includes a CPU 261 and a ROM 262, a RAM 264, a storage 265, an input device 266, a display device 267, and an external I / F 268. The data bus 263 is a communication line that connects these devices in the tamper detection server 103 to each other.

The CPU 261 is a controller for controlling the entire tampering detection server 103. The CPU 261 boots the OS of the tampering detection server 103 by executing a boot program stored in advance in the ROM 262, which is a non-volatile memory. The CPU 261 executes a tampering detection program stored in the storage 265 on this OS. The falsification detection program is a program for detecting a falsified portion included in a target manuscript by using a scanned image (that is, a processing target image) of the target manuscript acquired from a client device (for example, an image processing device 101). The RAM 264 is used as the main storage device of the CPU 261 and provides the CPU 261 with a temporary storage area (ie, work area).

The storage 265 may be a writable and readable auxiliary storage device, including a non-volatile memory, such as an HDD or SSD. The storage 265 stores various data such as the above-mentioned program, image data, and detection result data. The input device 266 includes, for example, a mouse and a keyboard, and accepts user input such as an operation instruction or information input from the user. The display device 267 includes, for example, an LCD or a CRT, and displays an image generated by the CPU 261 on the screen. The external I / F 268 transmits / receives data related to tampering detection to / from an external device via the network 105. The external I / F 268 may, for example, receive the image 21 to be processed from the image processing device 101 and transmit the detection result data 32 to the image processing device 101. Further, the external I / F 268 can transmit a request for providing the trained model 41 to the learning device 102 and receive the trained model 41 from the learning device 102. Further, the external I / F 268 can transmit a request for execution of the OCR to the OCR server 104, and can receive the recognition result data 31 indicating the result of the OCR from the OCR server 104.

Although not shown in FIG. 2, the configuration of the OCR server 104 may be the same as that of the tampering detection server 103.

<< 3. Process flow >>
FIG. 3A is a sequence diagram showing an example of a schematic flow of processing in the learning stage in the image processing system 100. FIG. 3B is a sequence diagram showing an example of a schematic flow of processing in the falsification detection stage in the image processing system 100. In the following description, the processing step is abbreviated as S (step).

<3-1. Learning stage ＞
In the learning stage, first, in S301, the operator sets the learning manuscript in which characters are written by hand in the image processing device 101, and instructs the image processing device 101 to read the manuscript. At this time, the operator inputs to the image processing device 101 via the input device 209 that the set learning manuscript is the original (not tampered with). The reading unit 111 of the image processing device 101 reads the set learning document in S302 in response to an instruction from the operator, and generates a scanned image 12. Further, the reading unit 111 adds a flag to the reading image 12 indicating that the reading image 12 is a learning original image. Further, in S303, the operator sets the falsified learning document in the image processing device 101, and instructs the image processing device 101 to read the document. At this time, the operator inputs to the image processing device 101 via the input device 209 that the set learning document includes a falsified portion. The reading unit 111 of the image processing device 101 reads the set learning document in S304 in response to an instruction from the operator, and generates a scanned image 13. Further, the reading unit 111 gives the read image 13 a flag indicating that the read image 13 is a falsified image for learning. By reading the identification information of the learning manuscript in S302 and S304, the reading unit 111 determines that the learning original image 12 and the learning falsified image 13 are a pair of the original learning manuscript and the falsified version. recognize. Then, the reading unit 111 associates the recognized document ID that identifies the learning manuscript with the learning original image 12 and the learning falsified image 13. Further, the reading unit 111 associates the data set ID that identifies the unit of generation / update of the learned model with the learning original image 12 and the learning falsified image 13. As an example, when one trained model is generated / updated for each image processing device, the data set ID may be an identifier that uniquely identifies the image processing device 101. As another example, when generating / updating one trained model for each user, the dataset ID may be an identifier that uniquely identifies each user. As another example, when one trained model is generated / updated for each user group, the data set ID may be an identifier that uniquely identifies each user group. In S305, the reading unit 111 sends the learning image and related data (that is, the learning original image 12, the learning falsified image 13, the flag, the document ID, and the data set ID) generated in this way to the learning device 102. Send.

FIG. 4 is a schematic diagram showing an example of a blank learning manuscript. In the example of FIG. 4, the learning manuscript 401 is a form-type document having eight character entry fields 402. The operator can create an original learning manuscript by entering arbitrary characters in each character entry field 402. The learning manuscript 401 has identification information 403 in the upper right. The identification information (also referred to as embedded information) 403 visually represents the document ID that uniquely identifies the learning manuscript 401 with a two-dimensional code. The position and size of the character entry field in the manuscript for the learning manuscript identified by each document ID are defined in advance and shared with the learning device 102. The document ID may be, for example, a UUID (Universally Unique Identifier).

FIG. 5 is a schematic view showing an example of a GUI (Graphical User Interface) for the image processing device 101 to receive an instruction for reading a document. The operation window 500 shown in FIG. 5 may be displayed on the screen of the display device 210, for example. The operation window 500 has a preview area 501, type designation buttons 502 and 503, a scan button 505, and a transmission start button 506. The type designation buttons 502 and 503 are buttons for causing the operator to specify whether the original or the falsified version is the type (also referred to as an attribute) of the original to be read or the scanned original. When the type designation button 502 is operated (for example, tapped), the reading unit 111 gives the read image a flag indicating that the read image is a falsified image for learning. On the other hand, when the type designation button 503 is operated, the reading unit 111 gives the read image a flag indicating that the read image is the original image for learning. In the operation window 500, the buttons corresponding to the specified type (original or tampered with) may be highlighted. The scan button 505 is a button for triggering the reading of the document set in the image processing device 101. When the scan button 505 is operated and the scan is completed, the preview image of the scanned image is displayed in the preview area 501. Before the data transmission is started, the operator sets another document in the image processing device 101 and operates the scan button 505 again to collect a plurality of scanned images and related data into the image processing device 101. May be retained by. When the scanning of at least one learning document is completed and the type of the document is specified, the transmission start button 506 can be operated. The transmission start button 506 is a button for triggering the transmission of the scanned image and related data. When the transmission start button 506 is operated, the reading unit 111 transmits the learning image and related data to the learning device 102 (see S305 in FIG. 3).

In S305 of FIG. 3A, the data processing unit 121 of the learning device 102 receives the learning image and related data from the image processing device 101. In S306, the data processing unit 121 stores the received learning image and related data in the storage unit 123. The data processing unit 121 and the learning unit 122 start processing for machine learning when a sufficient amount of data for learning is accumulated. First, in S307, the data processing unit 121 reads a pair of the learning original image 12 and the learning falsified image 13 from the storage unit 123, and generates learning data from these read images.

FIG. 6 is an explanatory diagram for explaining how the learning data is generated from the learning image. 6 (A) and 6 (B) show examples of the original learning image 12 and the falsified learning image 13, respectively. As described above, the learning original image 12 and the learning falsified image 13 constituting one pair have a common character entry field. In the example of FIG. 6, the number "1" is entered in the character entry field 402a of the learning original image 12. On the other hand, the content of the character entry field 402b of the falsified image 13 for learning has been falsified, and as a result of adding strokes, the character entry field 402b seems to indicate the number "4". The data processing unit 121 cuts out a partial image of the area of each character entry field from the learning original image 12 and the learning tampered image 13, and calculates a difference image between the partial images cut out from the common character entry field. The data processing unit 121 can know the position of each character entry field to be cut out based on the document ID acquired by reading the identification information 403. In this difference image, a pixel showing an absolute value of zero or a threshold value or less determined in advance is assumed not to belong to the falsified portion, and a pixel showing an absolute value exceeding the threshold value can be assumed to belong to the falsified portion. The character area image 611 shown in FIG. 6C is a partial image cut out from the character entry field 402b of the learning falsified image 13. The character area image 611 is an input image for learning that is input to the machine learning model by the learning unit 122. The binary image 612 shown in FIG. 6C is a partial image cut out from the character entry field 402a of the original image 12 for learning from the character area image 611 (after alignment as necessary). It is an image generated / updated by subtraction and further binarization based on the above threshold value. In the binary image 612, the pixel indicating true (white pixel in the figure) belongs to the falsified portion, and the pixel indicating false (black pixel in the figure) does not belong to the falsified portion. The binary image 612 is a teacher image that is treated as teacher data for machine learning by the learning unit 122.

The learning image may include a learning original image 12 without the corresponding learning falsified image 13. In this case, the data processing unit 121 indicates that the partial image cut out from the character entry field 402a of the original learning image 12 is the input image for learning, and all the pixels have the same size and are false (that is, there is no tampered portion as a whole). A binary image can be generated as a teacher image.

The data processing unit 121 generates a plurality of the above-mentioned input images and corresponding teacher images based on a plurality of pairs of the learning original image 12 and the learning tampered image 13 associated with the same data set ID. Next, in S308, the learning unit 122 repeatedly executes a learning process using these input images and teacher images within the same data set ID range, and generates a trained model 41 for tampering detection / Update. The trained model 41 may be, for example, a fully convoluted network (FCN) type model, although it is not limited. One iteration of the training process is, for example, inputting an input image into the model, calculating an error in the teacher data of the output data calculated according to the model (having provisional parameter values), and reducing the error. It may include adjustment of the parameter value of. For example, cross entropy can be used as an index of error. Further, as a method for adjusting the parameter value, for example, a back propagation method can be used. The learning unit 122 may repeat the learning process until it is determined that the learning has converged or the number of iterations reaches the upper limit. Then, the learning unit 122 stores the generated / updated learned model 41 (a set of model parameters constituting the learned model 41) in the storage unit 123 in association with the corresponding data set ID. The learning unit 122 may generate / update a separate trained model 41 for each of two or more different data set IDs. Further, the learning unit 122 may update the learned model 41 generated / updated in the past through additional learning processing using the newly acquired learning image. Further, the learning unit 122 may select learning data to be input to the learning process by any method of online learning, batch learning, or mini-batch learning.

<3-2. Falsification detection stage>
(1) Outline Flow In the tampering detection stage, first, in S351, the user sets the target document in the image processing device 101 and instructs the image processing device 101 to read the document. The user here may be the same person as the operator involved in the learning stage, or may be a different person. The scanning unit 111 of the image processing device 101 reads the set target document in S352 in response to an instruction from the user, and generates a scanned image 21. In addition, the user instructs the image processing device 101 to detect falsification of the target document in S353. In response to the instruction for tampering detection, the reading unit 111 gives the read image 21 a flag indicating that the read image 21 is the image to be processed in S354, and sets setting data related to tampering detection (for example, from the memory). )get. The setting data acquired here may include, for example, a data set ID (for example, an image processing device 101, an identifier that identifies a user or a user group) that identifies a trained model to be used for tampering detection. Then, in S355, the reading unit 111 transmits the processing target image 21 and the related data to the tampering detection server 103 together with the tampering detection request.

In S355, the image acquisition unit 131 of the tampering detection server 103 receives the tampering detection request and the processing target image 21 which is a scanned image of the target document and related data from the image processing device 101. The image acquisition unit 131 outputs the received image and data to the detection unit 132. In S356, the detection unit 132 requests the learning device 102 to provide the latest trained model 41. The model request transmitted here to the learning device 102 may include, for example, a dataset ID. The learning unit 122 of the learning device 102 reads the latest learned model 41 (specified by, for example, the data set ID) from the storage unit 123 in S357 in response to the reception of the model request, and the learned model 41 is read out. Is transmitted to the detection unit 132. Next, in S358, the detection unit 132 transmits the processing target image 21 to the OCR server 104, and requests the OCR server 104 to recognize the characters included in the processing target image 21. In response to the reception of the OCR request, the character recognition unit 141 of the OCR server 104 executes OCR on the processing target image 21 in S359 to recognize the character and the position of the character area in the processing target image 21. Next, the character recognition unit 141 transmits the recognition result data 31 indicating the recognition result to the detection unit 132 in S360. The detection unit 132 detects the falsified portion included in the target manuscript by applying the processing target image 21 to the learned model 41 provided by the learning device 102 in S361. The trained model 41 is a model generated / updated through machine learning using a learning image which is a scanned image of a learning manuscript as described above. In the tampering detection process here, the detection unit 132 applies a partial image of the process target image 21 to the process target image 21 for each character region recognized as a result of OCR, for example. As a result, a falsification detection result, which is a bitmap indicating whether or not each pixel belongs to the falsification portion, is generated for each of the character regions of the characters recognized in the processing target image 21. Next, the detection unit 132 transmits the detection result data 32 to the image processing device 101 in S362. The detection result data 32 is integrated with the same size as the processing target image 21 (indicating whether or not each pixel belongs to the falsified portion) generated by integrating the above-mentioned bitmaps obtained for each character area. Includes bitmaps. This integrated bitmap is referred to as a detection result image in the following description. The detection unit 132 additionally generates an enhanced image showing the pixels (hereinafter referred to as tampered pixels) determined to belong to the tampered portion as a result of the tampering detection in the processed image 21, and generates the generated enhanced image. It may be included in the detection result data 32.

FIG. 7A shows an image to be processed 21a as an example. FIG. 7B shows a detection result image 32a showing a falsification detection result for the processing target image 21a. FIG. 7C shows an emphasized image 32b in which the falsified portion of the image to be processed 21a is emphasized. The processing target image 21a is an image generated by reading the contract as the target manuscript. The image to be processed 21a has a plurality of character areas, and one of the character areas 701a appears to indicate the number "4". These character areas can be recognized as a result of OCR executed by the character recognition unit 141 of the OCR server 104. The detection unit 132 of the tampering detection server 103 cuts out a character area image of each character area from the processing target image 21, and applies the trained model 41 to each character area image. As a result, it is determined whether or not each pixel in each character area belongs to the falsified portion. The detection result image 32a is a binary image in which the falsification detection result is integrated as an entire image. In the detection result image 32a, the pixel determined to belong to the falsified portion indicates true (black in the figure), and the pixel determined not to belong to the falsified portion indicates false (white in the figure). For example, the pixels that make up some strokes of the number "4" in the character area 701a indicate true, which means that the strokes may have been added ex post facto for tampering. To do. The emphasized image 32b is an image generated by changing the color of the pixel in which the detection result image 32a indicates true in the processing target image 21a to a specific color. The specific color here may be, for example, red (RGB = (255,0,0)), although it is not limited. In the example of FIG. 7C, the color of a part of the stroke of the number “4” is changed in the character area 701a in the emphasized image 32b. The method of emphasizing the pixels determined to belong to the falsified portion is not limited to the color change as described above, and may be any method such as thickening or blinking.

In S362 of FIG. 3, the display control unit 112 of the image processing device 101 receives the above-mentioned detection result data 32 from the tampering detection server 103. In S363, the display control unit 112 displays the result of tampering detection on the screen based on the detection result data 32. In particular, in the present embodiment, when the detection result data 32 indicates that a falsified portion has been detected in the target document, the display control unit 112 compares the emphasized image and the comparative image of the region including the falsified portion on the screen. To display. The emphasized image here is an image in which the falsified portion of the processed image 21 is emphasized, and the comparative image is an image showing the falsified portion as in the processed image 21. As a result, the user can grasp which part of the image is suspected of being tampered with from the emphasized image, and then look at the color or shade of the part in the comparison image to determine whether or not the tampering has actually occurred. it can. An example of such a contrasting display, and some variations of the comparative image, will be further described later.

(2) Specific flow of tampering detection stage (image processing device)
FIG. 8 is a flowchart showing an example of a specific flow of processing executed by the image processing device 101 in the tampering detection stage. The process shown in FIG. 8 is executed in the image processing device 101 under the control of the CPU 201 that executes the controller program loaded from the storage 208 to the RAM 204. The process may be started when a predetermined operation by the user is detected via the input device 209 of the image processing device 101.

First, in S801, the scanning unit 111 reads the target document set in the transport device 207 with the scanner device 206 to generate a processing target image. The image to be processed may be, for example, a full-color (RGB3 channel) image. Further, the reading unit 111 receives a tampering detection instruction input by the user via the input device 209 in S802. Next, in S803, the reading unit 111 transmits the image to be processed and the related data (for example, the data set ID) to the tampering detection server 103 via the external I / F 211 together with the tampering detection request. After that, the display control unit 112 waits for the reception of the detection result data from the tampering detection server 103 in S804. When the detection result data is received from the tampering detection server 103 via the external I / F 211, the process proceeds to S805. In S805, the display control unit 112 determines whether the detection result data indicates that the target document includes a falsified portion. If it is determined that the target manuscript includes a falsified portion, the process proceeds to S806. On the other hand, if it is determined that the target document does not include the falsified portion, the process proceeds to S810. In S806, the display control unit 112 determines whether the emphasized image and the comparative image should be displayed in contrast with each other or only the emphasized image should be displayed as the tampering detection result based on the user input. In the former case, the process proceeds to S807, and in the latter case, the process proceeds to S808. In S807, the display control unit 112 displays a emphasized image in which the falsified portion of the processed image is emphasized and a comparative image in which the falsified portion is shown as the processed image in comparison with each other on the screen of the display device 210. Let me. An example of the contrasting display here will be further described later with reference to FIGS. 13 (A) and 13 (B) and FIG. In S808, the display control unit 112 causes the display device 210 to display a highlighted image that emphasizes the falsified portion of the image to be processed. An example of the display here will be further described later with reference to FIGS. 11 (B) and 12. In S810, the display control unit 112 displays on the screen of the display device 210 that no tampering has been detected with respect to the image to be processed. In addition to displaying such a tampering detection result, the image processing device 101 may store the image to be processed, the emphasized image, and the comparison image as image data in the storage 208, or the image processing device 101 may store the image to be processed, or the image processing device 101 may be stored as image data. It may be transmitted to the device of. Further, one or more of these images may be printed by the printer device 205.

(3) Specific flow of tampering detection stage (tampering detection server)
FIG. 9 is a flowchart showing an example of a specific flow of processing executed by the tampering detection server 103 in the tampering detection stage. The process shown in FIG. 9 is executed by the tamper detection server 103 under the control of the CPU 261 that executes the controller program loaded from the storage 265 to the RAM 264. The process can be started when the tampering detection request is received from the image processing device 101 via the external I / F 268 (the standby of the tampering detection request can be started in response to the power-on of the tampering detection server 103). ..

First, in S901, the image acquisition unit 131 receives the image to be processed and related data (for example, data set ID) from the image processing device 101 via the external I / F 268 together with the falsification detection request. Next, in S902, the detection unit 132 transmits a request for providing the trained model to the learning device 102 via the external I / F 268, and acquires the trained model from the learning device 102. Here, the detection unit 132 acquires, for example, a trained model specified by the data set ID received together with the tampering detection request. For example, the detection unit 132 constructs a neural network model on the RAM 264 and reflects the value of the model parameter received from the learning device 102 in the constructed model. Further, in S903, the detection unit 132 transmits a request for execution of OCR for the image to be processed to the OCR server 104 via the external I / F 268 together with the image to be processed, and OCR transmits the recognition result data indicating the result of OCR. Receive from server 104. Next, in S904, the detection unit 132 cuts out a character area image of one of the characters recognized in the processing target image from the processing target image, and applies the cut out character area image to the trained model acquired in S902. .. As a result, the detection unit 132 determines whether or not each of the plurality of pixels in the character area image belongs to the falsified portion. The text area image may be grayscaled prior to application to the trained model. The result of the determination here is a bitmap similar to the binary image 612 shown in FIG. 6 (C). Next, the detection unit 132 determines whether or not there is a remaining character area in the image to be processed. If there is a remaining character area, the detection unit 132 repeats the determination of S904 for the next character area. If no unprocessed character area is left in the image to be processed, the process proceeds to S906. In S906, the detection unit 132 integrates the detection results obtained for each character area as a result of the repetition of S904 into one bitmap to generate a detection result image. Further, in S907, the detection unit 132 generates an emphasized image in which the falsified portion in the image to be processed is emphasized. Then, in S908, the detection unit 132 transmits the detection result data including the detection result image and the emphasized image to the image processing device 101 via the external I / F 268.

Although an example in which the detection unit 132 cuts out a character region image from the image to be processed based on the result of OCR has been described here, OCR may not always be used. For example, when the target manuscript is a form having a known format, the detection unit 132 processes an image of a partial region (for example, a rectangle) at a predetermined position as a character region image according to the known format. It can be cut out from the target image.

<< 4. Details of display control >>
FIG. 10 is a schematic diagram showing an example of a GUI for setting and instructing tampering detection that can be displayed on the screen of the display device 210 under the control of the display control unit 112 of the image processing device 101. The setting window 1000 shown in FIG. 10 has a preview area 1001, an emphasis setting button 1002, a file name change setting button 1003, a folder distribution setting button 1004, a scan button 1005, and a transmission start button 1006. The emphasis setting button 1002 is a button for setting whether or not to enable emphasis on the tampered portion when displaying the tampering detection result. The file name change setting button 1003 is a button for setting whether or not to enable automatic change of the file name of the scanned image data based on the tampering detection result. The folder distribution setting button 1004 is a button for setting whether or not to enable automatic folder distribution of the scanned image data based on the tampering detection result. In the settings window 1000, the buttons corresponding to the activated settings may be highlighted. The scan button 1005 is a button for triggering the reading of the document set in the image processing device 101. When the scan button 1005 is operated and the scan is completed, the preview image of the scanned image is displayed in the preview area 1001. Before the data transmission is started, the user sets another document in the image processing device 101 and operates the scan button 1005 again to collect a plurality of scanned images and related data in the image processing device 101. May be retained by. When the scanning of at least one target document is completed, the transmission start button 1006 can be operated. The transmission start button 1006 is a button for inputting a tampering detection instruction that triggers transmission of the image to be processed and related data. When the transmission start button 1006 is operated, the reading unit 111 transmits the image to be processed and the related data to the tampering detection server 103 together with the tampering detection request.

11 (A) and 11 (B) are examples of GUIs for displaying a list and details of tampering detection results that can be displayed on the screen of the display device 210 under the control of the display control unit 112. It is a schematic diagram which shows. The list window 1100 shown in FIG. 11A has a list area 1101, a file name change button 1103, a folder distribution button 1104, and an OK button 1105. In the list area 1101, three list items 1102a, 1102b, and 1102c corresponding to the falsification detection results of the three target originals are listed. Each list item has a "manuscript ID" that identifies the target manuscript (or the corresponding scanned image), a "date and time" that indicates when tampering detection was executed, and a "tampering" that indicates whether or not tampering was detected in the manuscript. Presence or absence ”is included as a data item. A preview of the scanned image is displayed at the right end of each list item. In the example of FIG. 11A, the two target manuscripts having the manuscript ID = "Scan1" and "Scan3" are determined to include the tampered portion as a result of tampering detection, while the manuscript ID = "Scan2". It is determined that the target manuscript is not included in the falsified part. The display control unit 112 can determine that the target manuscript in which one or more pixels indicated to belong to the falsified portion in the detection result image included in the detection result data 32 is present is “falsified”. When, for example, the list item 1102a is operated (for example, tapped) by the user, the display control unit 112 displays a detail window 1150 for the target document identified by the document ID = "Scan1" on the screen. The file name change button 1103 is a button for triggering the execution of automatic change of the file name of the scanned image data based on the tampering detection result. When the file name change button 1103 is operated by the user, the display control unit 112 automatically changes the file name of the scanned image data corresponding to the target document in which each check box is checked. For example, a predetermined prefix or suffix (for example, "tampered with") indicating that there is falsification may be added to the file name of the read image data of "tampered with". A predetermined prefix or suffix (for example, "no tampering") indicating that there is no tampering may be added to the file name of the read image data of "no tampering". The folder distribution button 1104 is a button for triggering the execution of automatic folder distribution of the read image data based on the tampering detection result. When the folder distribution button 1104 is operated by the user, the display control unit 112 automatically distributes the scanned image data corresponding to the target document in which each check box is checked to any of a plurality of folders. For example, the “tampered” scanned image data may be stored in a first folder (eg, having the folder name “tampered”). Further, the read image data of "no tampering" can be stored in a second folder (for example, having a folder name of "no tampering"). When the OK button 1105 is operated by the user, the display control unit 112 ends the display of the list window 1100.

The detail window 1150 shown in FIG. 11B has an image display area 1151, a correction button 1153, a comparison display button 1154, a transmission button 1155, a print button 1156, and an OK button 1157. The image display area 1151 is an area for displaying the emphasized image of the target original specified in the list window 1100. In the example of FIG. 11B, the emphasized image 32b is displayed in the image display area 1151. When the correction button 1153 is operated by the user, the display control unit 112 displays a correction window 1200, which will be described later, on the screen, which enables the user to correct the tampering detection result. Further, when the contrast display button 1154 is operated, the display control unit 112 causes the contrast display window 1300 or 1350, which will be described later, in which the emphasized image and the comparison image are arranged in contrast to be displayed on the screen. The comparison display button 1154 may be operable only for the target document that is determined to include the tampered portion as a result of the tampering detection. Further, when the transmission button 1155 is operated, the display control unit 112 transmits one or both of the processed target image and the emphasized image of the designated target document to the other device. The transmission of the image here may be performed by any method, for example, attachment to an e-mail, or transmission of an e-mail containing a link on a file server or other message. The destination of transmission may be a destination registered in advance in the image processing device 101, or may be specified by the user via a pop-up destination window (not shown). Further, when the print button 1156 is operated, the display control unit 112 causes the printer device 205 to print one or both of the processed target image and the emphasized image of the designated target document. Further, when the OK button 1157 is operated, the display control unit 112 ends the display of the detail window 1150 and causes the list window 1100 to be displayed on the screen again.

FIG. 12 is a schematic diagram showing an example of a GUI that can be displayed on the screen of the display device 210 under the control of the display control unit 112 and that allows the user to modify the tampering detection result. The modification window 1200 shown in FIG. 12 has an image display area 1201, a tampered pixel designation button 1203, a tampered pixel release button 1204, and an OK button 1205. The image display area 1201 is an area for displaying the emphasized image of the designated target document. In the example of FIG. 12, the emphasized image 32b is displayed in the image display area 1201. When the falsification pixel designation button 1203 is operated by the user, the display control unit 112 changes the pixel designated by the user in the image display area 1201 to a pixel belonging to the falsification portion in the falsification detection result. Further, when the falsification pixel release button 1204 is operated by the user, the display control unit 112 changes the pixel specified by the user in the image display area 1201 to a pixel that does not belong to the falsification portion in the falsification detection result. When the OK button 1205 is operated, the display control unit 112 reflects the correction of the tampering detection result by the user in the detection result data 32, and ends the display of the correction window 1200. In the list window 1100 of FIG. 11 (A), the automatic change of the file name or the automatic folder distribution triggered by the operation of the button 1103 or 1104 is the modified tampering detection when the tampering detection result is modified by the user in this way. It can be executed based on the result. As a result, it is possible to more accurately process human or systematic documents (using the folder name or storage folder as a clue) following the verification of the presence or absence of tampering based on the tampering detection result after modification. It becomes.

When the tampering detection result is modified by the user via the modification window 1200 described above, the display control unit 112 may update the trained model based on the modified tampering detection result. Specifically, the display control unit 112 transmits, for example, a pair of the processing target image and the corrected detection result image to the learning device 102 together with the model update request. The learning unit 122 of the learning device 102 receives a model update request from the display control unit 112, inputs a character area image of each character area in the processing target image, and a character in the same character area in the detection result image. The trained model can be updated using the region image as the teacher image. As a result, the pixel pattern that the current trained model tends to make an error in detection is relearned, so that the accuracy of the tampering detection of the trained model can be effectively improved.

FIG. 13A is a schematic diagram showing a first example of a GUI in which a emphasized image and a comparative image are arranged in contrast, which can be displayed on the screen of the display device 210 under the control of the display control unit 112. is there. In the first example, an emphasized image and a comparative image showing the entire target document are displayed. The contrast display window 1300 shown in FIG. 13A has a highlighted image display area 1301, a comparative image display area 1302, a character-specific display button 1303, a correction button 1305, and an OK button 1306. In the example of FIG. 13A, the highlighted image 32b is displayed in the highlighted image display area 1301 and the comparison image 21a is displayed in the comparison image display area 1302. The emphasized image 32b is an image in which the pixels determined to belong to the falsified portion in the scanned image of the target document are emphasized and shown. The comparative image 21a is an image that is the same as the image to be processed 21a and shows the pixels determined to belong to the falsified portion as the read image (without emphasizing). When the character-specific display button 1303 is operated by the user, the display control unit 112 shifts the screen to the contrast display window 1350, which will be described later, in which the emphasized image and the comparison image cut out for each character area are arranged in contrast. .. Further, when the correction button 1305 is operated, the display control unit 112 displays on the screen the above-mentioned correction window 1200 that enables the user to correct the tampering detection result. Further, the display control unit 112 ends the comparison display when the OK button 1306 is operated.

FIG. 13B is a schematic diagram showing a second example of a GUI in which a emphasized image and a comparative image are arranged in contrast, which can be displayed on the screen of the display device 210 under the control of the display control unit 112. is there. In the second example, a emphasized image and a comparison image, which are partial images (character area images) for each character, are displayed. The contrast display window 1350 shown in FIG. 13B has a list area 1351, an overall display button 1353, a comparison mode switching button 1354, and an OK button 1356. In the list area 1351, character areas of one or more characters recognized in the target document are listed in a form that can be scrolled in the vertical direction. In the example of FIG. 13B, two list items 1352a and 1352b are displayed in the list area 1351. Each list item can be uniquely identified by the combination of the manuscript ID of the target manuscript and the number (No.) assigned to each character area. The list area 1351 has a highlighted image display area 1361 in the center and a comparative image display area 1362 at the right end. In each emphasized image display area 1361, a partial image of one character area in the emphasized image including the emphasized falsified portion (here, this partial image is also referred to as an emphasized image) is displayed. In each comparative image display area 1362, a partial image of one character area in the image to be processed (here, this partial image is also referred to as a comparative image) including a falsified portion that is not emphasized is displayed. However, the content of the comparison image displayed in the comparison image display area 1362 may change depending on the comparison mode described later. When the highlighted image display area 1361 or the comparative image display area 1362 is operated (for example, tapped) by the user, the display control unit 112 enables the user to correct the tampering detection result for each character area, which will be described later. Display the correction window 1500 on the screen. Further, when the entire display button 1353 is operated by the user, the display control unit 112 shifts the screen to the comparison display window 1300 described above. Further, when the comparison mode switching button 1354 is operated, the display control unit 112 switches the comparison mode setting depending on the user input, as will be described later. Further, the display control unit 112 ends the comparison display when the OK button 1356 is operated.

The display control unit 112 may support only a single comparison mode, or may dynamically switch the comparison mode for comparison display among a plurality of comparison mode candidates. In the former case, the comparison display window 1350 does not have to have the comparison mode switching button 1354. In the latter case, the comparison mode candidates may include, for example, two or more of the following:
-Comparison mode C1: The comparison image is an image of a character region including an unemphasized falsified portion.
-Comparison mode C2: The comparison image is an image including a character area including an unemphasized falsified part and a peripheral area of the character area.
Comparison mode C3: The comparison image includes an image of another character area showing the same characters as the characters shown in the highlighted image.
-Comparison mode C4: The comparison image is an image of a character region including a falsified portion that has been suppressed or hidden.

The display control unit 112 may display a list of candidates for these comparison modes on the screen and allow the user to specify a desired comparison mode. Instead, the comparison mode settings may be toggled (switched in sequence) in a predetermined order each time the comparison mode switching button 1354 is operated. The display control unit 112 can switch the content of the comparison image displayed in the comparison image display area 1362 of the comparison display window 1350 according to the comparison mode thus designated by the user.

FIG. 14A shows an example of a comparison image in the comparison mode C1. The comparative image 1401 shown in FIG. 14A is an image cut out from the processing target image at the same position and the same size as the character area of the character indicated by the corresponding emphasized image. In the comparative image 1401, the falsified portion is shown as the scanned image generated by scanning the target document without being emphasized. In the comparison mode C1, the user can clearly grasp the correspondence between the parts constituting the characters between the emphasized image and the comparison image.

FIG. 14B shows an example of a comparison image in the comparison mode C2. The comparative image 1402 shown in FIG. 14B is an image cut out from the image to be processed so as to include the peripheral region of the comparative image 1401. Also in the comparative image 1402, the falsified portion is shown as the scanned image generated by scanning the target document without being emphasized. In the comparison mode C2, the user can evaluate factors such as color, shading, and stroke habit through the characters before and after the character suspected of being tampered with, and judge the validity of the tampering detection result.

FIG. 14C shows an example of a comparison image in the comparison mode C3. In the comparison mode C3, for example, the comparison image 1401 and the comparison image 1403 can be displayed in the comparison image display area 1362 of the comparison display window 1350. The comparative image 1403 is an image of another character area showing the same characters as the characters shown in the character area of the corresponding emphasized image. The display control unit 112 can determine, for example, whether or not two characters written at different positions are the same character based on the OCR result. Also in the comparative image 1403, the falsified portion is shown as the scanned image generated by scanning the target document without being emphasized. In the comparison mode C3, the user can determine the validity of the tampering detection result by referring to factors such as the color, shading, and stroke habit of the same character as the character suspected of being tampered with.

FIG. 14D shows an example of a comparison image in the comparison mode C4. The comparison image 1404 shown in FIG. 14 (D) is an image cut out from the processing target image at the same position and the same size as the character area of the character indicated by the corresponding emphasized image. However, unlike the comparative image 1401, in the comparative image 1404, the falsified portion is suppressed or hidden. In the comparison mode C4, the user can see the contents of the document without strokes that may have been added by tampering, and pay attention to the pixels that are determined not to belong to the tampering part. The validity can be judged.

FIG. 15 is a schematic diagram showing another example of a GUI that can be displayed on the screen of the display device 210 under the control of the display control unit 112 and that allows the user to modify the tampering detection result. The modification window 1500 shown in FIG. 15 has a highlighted image display area 1501, a comparison image display area 1502, a comparison mode switching button 1503, a tampered pixel designation button 1504, a tampered pixel release button 1505, and an OK button 1506. The highlighted image display area 1501 is an area for displaying a highlighted image of the character area of the designated target document. In the example of FIG. 15, the highlighted image 1511 is displayed in the highlighted image display area 1501. The comparative image display area 1502 is an area for displaying a comparative image of the character area of the designated target document. In the example of FIG. 15, the comparison image 1512 is displayed in the comparison image display area 1502. Here, it is assumed that the above-mentioned comparison mode C1 is set. When the comparison mode switching button 1503 is operated, the display control unit 112 switches the comparison mode setting between the two or more comparison mode candidates described above, and displays a comparison image corresponding to the newly set comparison mode. It is displayed in the comparative image display area 1502. When the falsification pixel designation button 1504 is operated by the user, the display control unit 112 changes the pixel designated by the user in the highlighted image display area 1501 to a pixel belonging to the falsification portion in the falsification detection result. Further, when the falsification pixel release button 1505 is operated by the user, the display control unit 112 changes the pixel specified by the user in the highlighted image display area 1501 to a pixel that does not belong to the falsification portion in the falsification detection result. When the OK button 1506 is operated, the display control unit 112 reflects the correction of the tampering detection result by the user in the detection result data 32, and ends the display of the correction window 1500.

FIG. 16 is a flowchart showing an example of a specific flow of display control processing executed by the image processing device 101 when falsification of the target document is detected. The process shown in FIG. 16 is executed in the image processing device 101 under the control of the CPU 201 that executes the controller program loaded from the storage 208 to the RAM 204. The process may correspond to S807 in FIG.

First, the display control unit 112 determines in S1601 whether the character-specific contrast display or the overall contrast display is designated as the display mode of the contrast display. For example, when the contrast display button 1154 of the detail window 1150 of FIG. 11 (B) or the entire display button 1353 of the contrast display window 1350 of FIG. 13 (B) is operated, the display control unit 112 specifies the overall comparison display. It can be determined that it has been done. On the other hand, when the character-specific display button 1303 of the comparison display window 1300 of FIG. 13A is operated, the display control unit 112 can determine that the character-specific comparison display is specified. If it is determined that the character-specific contrast display is specified, the process proceeds to S1602. On the other hand, when it is determined that the overall comparison display is specified, the process proceeds to S1620.

In S1602, the display control unit 112 acquires character area data indicating the position and size of one or more character areas in the image. For example, the display control unit 112 may receive the recognition result data 31 indicating the result of character recognition by the OCR server 104 as the character area data from the tampering detection server 103 together with the detection result data 32. Instead, when the target manuscript is a form having a known format, the display control unit 112 acquires the character area data in which the position and size of the character area included in the known format are defined in advance from the storage 208. You may. Subsequent steps S1603 to S1612 are repeated for each of the character areas including the pixels determined to belong to the falsified portion by the detection result data 32 in the character area indicated by the character area data. Taking the processing target image 21a and the detection result image 32a of FIG. 7A as an example, the character area 701a and the rectangular character area of the two characters to the right of the character area 701a are pixels determined to belong to the falsified portion. The character area to be included. Therefore, S1603 to S1612 can be repeated for these three character areas.

In S1603, the display control unit 112 selects one of the character areas including the falsified part (hereinafter, the selected character area is referred to as a selection area). Next, in S1604, the display control unit 112 cuts out a partial image of the selected area from the emphasized image according to the position and size indicated by the character area data. Subsequent processing branches in S1605 depending on whether or not the comparison mode currently set is the comparison mode C2. If the comparison mode C2 is currently set, the process proceeds to S1606. When the comparison mode C1, C3 or C4 is set, the process proceeds to S1607. In the case of the comparison mode C2, the display control unit 112 cuts out a partial image including the selected area and the peripheral area outside the selected area from the scanned image in S1606. As an example, the size of the partial image including the peripheral area is W times in the horizontal direction and H times in the vertical direction with respect to the size of the selected area (W and H are magnifications larger than the preset 1; for example, W = 4). And H = 2). As another example, the peripheral area may be dynamically set as an area containing N characters (N is a preset integer) in the vicinity of the selected area. In the comparison mode C2, the partial image cut out here becomes the comparison image. On the other hand, in the case of the comparison mode C1, C3 or C4, the display control unit 112 cuts out a partial image of the selected area from the scanned image in S1607. When the comparison mode C1 is set at the present time, the partial image cut out in S1607 becomes the comparison image. If the comparison mode C3 is currently set, the process proceeds to S1609 at the branch of S1608. In S1609, the display control unit 112 cuts out a partial image of one or more other character areas showing the same character as the character indicated by the selected area from the scanned image. It is desirable that the other character area here is an area that does not include pixels determined to belong to the falsified portion. The display control unit 112 requests OCR from the OCR server 104 here, and identifies another character area indicating the same character as the character indicated by the selection area based on the result of character recognition returned from the OCR server 104. You may. If such another character area does not exist in the target manuscript, S1609 may be skipped. In the comparison mode C3, the combination of the partial images cut out in S1607 and S1609 (for example, an image in which the two partial images are arranged side by side) becomes the comparison image. If the comparison mode C4 is currently set, the process proceeds to S1611 through the branching of S1608 and S1610. In S1611, the display control unit 112 suppresses the value of the pixel determined to belong to the falsified portion in the partial image cut out in S1607 (for example, corrects the pixel value to the same color as the background color such as white or a color close to it). ). In the comparison mode C4, the partial image processed in this way becomes the comparison image. In S1612, the display control unit 112 determines whether or not an unprocessed character area including a pixel determined to belong to the falsified portion remains. When such an unprocessed character area is left, the process returns to S1603, and the above-mentioned S1603 to S1611 are repeated for the next character area. If not, the process proceeds to S1613.

In S1613, the display control unit 112 causes one or more pairs of the emphasized image and the comparison image for each character area to be displayed in contrast in the contrast display window. According to such a contrast display for each character, the user grasps from the highlighted image which part of each character is suspected of being tampered with, and then sees the color or shade of the part in the comparison image. Therefore, it can be determined whether or not each character has actually been tampered with.

In S1620, the display control unit 112 displays the emphasized image and the comparative image (scanned image) showing the entire target document in a contrasting manner in the contrasting display window. As described above, in the present embodiment, it is possible to smoothly switch between the contrast display showing the entire manuscript and the contrast display for each character described above. As a result, the user can pass through the target manuscript while going back and forth between a rough grasp of the character at which position in the manuscript is suspected of being tampered with in the overall display and verification of each character in the character-specific display. Verification of tampering detection results can be carried out efficiently.

Although FIG. 16 has described an example in which the display control unit 112 uses the emphasized image received from the tampering detection server 103, the display control unit 112 starts from the processing target image (read image) based on the tampering detection result. You may generate the emphasized image by yourself.

<< 5. Modification example >>
In the above-described embodiment, a method of determining whether or not each pixel in the character area belongs to the falsified part by applying the character area image of each character area to the trained model has been mainly described. However, the method for detecting tampering is not limited to such an example. For example, as a first modification, tampering detection may be performed for each character instead of each pixel. In the character-by-character tampering detection, for example, whether each character area includes a tampered portion instead of the teacher image (for example, the binary image 612 of FIG. 6C) indicating whether or not each pixel belongs to the tampered portion. A trained model generated / updated using a flag indicating whether or not is used as teacher data can be used. The trained model may be an arbitrary discriminant model that outputs a bit indicating whether or not a character area image (input image) contains a tampered part, and is, for example, a kind of neural network model. A VGG may be used.

FIG. 17 shows an example of a GUI in which the emphasized image and the comparison image are arranged in contrast based on the result of the falsification detection for each character in the first modification. The contrast display window 1700 shown in FIG. 17 has a highlighted image display area 1701, a comparative image display area 1702, a character-specific display button 1703, a correction button 1305, and an OK button 1306. In the example of FIG. 17, the emphasized image 1732 is displayed in the emphasized image display area 1701 and the comparative image 1721 is displayed in the comparative image display area 1702. The emphasized image 1732 is an image in which the characters in the character area determined to include the falsified portion in the scanned image of the target document are emphasized. The comparative image 1721 is the same image as the image to be processed 21a, and is an image showing the characters in the character area determined to include the falsified portion as the read image (without emphasizing). When the character-specific display button 1703 is operated by the user, the display control unit 112 shifts the screen to a contrast display window (not shown) in which the emphasized image and the comparison image cut out for each character area are arranged in contrast. You may let me.

As a second modification, the tampering detection uses an autoencoder type model (for example, VAE (Variational Autoencoder)) that encodes a character area image and extracts character features instead of a discriminant type model. May be done. The VAE learning process includes an encoder process for calculating a dimension-reduced feature amount from an input image and a decoder process for restoring an input image from the calculated feature amount. The encoder has the structure of a neural network, and the decoder has the structure opposite to that of the encoder. The model error is evaluated as the difference between the input image and the restored image (eg, cross entropy), and the values of the model parameters are adjusted, for example, by the backpropagation method so that the error is small. In this case, the falsified image for learning and the teacher data are unnecessary. In the learning stage, the learning device 102 generates / updates a learned model for extracting the feature amount of the character without tampering from the character area image through the learning process using a plurality of original images for learning. Typically, it is not practical to generate / update a single model capable of extracting appropriate features for every character, so the learning device 102 can learn different model parameter values for each character type. For example, characters such as "1", "2", "3", "two", and "ni" may be treated as different types of characters. In the tampering detection stage, the tampering detection server 103 applies each character area image to, for example, an encoder of a trained model corresponding to a character type recognized as a result of OCR, and extracts a feature amount from the character area image. Further, the tampering detection server 103 acquires the reference feature amount extracted in advance for the known (non-tampering) character area image of the same character type. Then, the tampering detection server 103 determines that the character area image does not include the tampering portion when the difference between the two feature amounts satisfies a predetermined condition (for example, when the Manhattan distance is equal to or less than a certain threshold value). obtain. On the contrary, when the Manhattan distance between the feature amount extracted from the character area image and the reference feature amount exceeds the above threshold value, it can be determined that the character area image includes a falsified portion. In this modification, as in the first modification, it is determined whether or not each of the characters recognized in the image to be processed includes a tampered portion (that is, tampering detection for each character). The contrast display of the emphasized image and the comparative image in this modified example may be performed in the same manner as in the example described with reference to FIG.

In the above-described embodiment, an example in which the highlighted image and the comparative image are displayed side by side in the horizontal direction in the contrast display window has been mainly described. However, the emphasized image and the comparative image may be displayed side by side in any direction other than the horizontal direction. In addition, the emphasized image and the comparison image may be displayed in separate windows. Further, as a third modification, the emphasized image and the comparative image may be displayed one after the other in time instead of being arranged in a spatially contrasting manner. For example, in a single image display area, the display of the emphasized image for X seconds and the display of the comparative image for X seconds may be alternately and repeatedly performed. The term "contrast indication" herein is intended to include all of these indication aspects.

<< 6. Summary >>
Up to this point, the embodiments of the present disclosure have been described in detail with reference to FIGS. 1 to 17. In the above-described embodiment, the falsified portion included in the target manuscript is detected by using the scanned image of the target manuscript, and the emphasized image showing the falsified portion in the scanned image with emphasis and the falsified portion of the scanned image are shown. The comparison images shown on the street are displayed in contrast on the screen. According to such a configuration, when the user verifies the tampering detection result, the emphasized image clearly shows the user which part of the image is suspected of being tampered with, and the original color of the part is shown in the comparison image. And shades can be presented to the user. Therefore, the user can easily determine the validity of the detection result from the tint or shade of the falsified portion after grasping the position of the falsified portion.

Further, in the above-described embodiment, the emphasized image and the comparison image can be displayed in contrast according to the comparison mode specified by the user from two or more comparison modes having different contents of the comparison image. According to the first comparison mode, the user can clearly grasp the correspondence between the parts constituting the characters between the emphasized image and the comparison image. According to the second comparison mode, the user can evaluate factors such as color, shading, and stroke habit through the characters before and after the character suspected of being tampered with, and judge the validity of the tampering detection result. According to the third comparison mode, the user can determine the validity of the tampering detection result by referring to factors such as the color, shading, and stroke habit of the same character (in other character areas) as the character suspected of being tampered with. You can judge. According to the fourth comparison mode, the user looks at the image without strokes that may have been added by tampering, and pays attention to the pixels determined not to belong to the tampering part, and the tampering detection result. Can be judged as valid. By enabling flexible switching between these comparison modes, it is possible to streamline the work of the user who verifies the validity of the tampering detection result.

Further, in the above-described embodiment, the user can be provided with a user interface capable of modifying the falsification detection result indicating which part of the scanned image is determined to be falsified. According to such a configuration, when the user discovers a detection error by looking at the contrasting display of the emphasized image and the comparison image, the user can immediately correct the detection error. Therefore, the tampering detection result appropriately modified by the user can be smoothly passed to the human or systematic process following the verification of the presence or absence of tampering.

In a certain embodiment, it is determined whether or not each of the plurality of pixels in the scanned image belongs to the falsified portion, and the pixel determined to belong to the falsified portion can be emphasized in the emphasized image. In this case, it is possible to visually present the fine tampering detection result to the user, for example, by emphasizing only the stroke in the case of tampering in which strokes are added to individual characters. In a modified example, it is determined whether or not each of one or more character areas in the scanned image includes a falsified portion, and the characters in the character area determined to include the falsified portion are emphasized in the emphasized image. obtain. In this case, since the load of the calculation process required for the tampering detection is small, the tampering detection result can be quickly presented to the user even when the amount of data of the scanned image is large.

<< 7. Other embodiments >>
In the above embodiment, a program that realizes one or more functions is supplied to a system or a device via a network or a storage medium, and one or more processors in the computer of the system or the device read and execute the program. It is also feasible in the form. It can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

The above embodiment may be realized in a system including a plurality of devices, or may be realized in one device as a whole. Further, the functions described above as being included in a single device may be distributed and arranged in a plurality of devices. For example, the data processing unit 121 and the learning unit 122 of the learning device 102 may be physically mounted on separate devices. Further, the image processing device 101 may be equipped with a function equivalent to that of the detection unit 132 of the tampering detection server 103. Similarly, a function equivalent to that of the character recognition unit 141 of the OCR server 104 may be implemented in the tampering detection server 103 or the image processing device 101. Any image referred to herein may be a grayscale image or a full color image.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

100 Image processing system 101 Image processing server 111 Reading unit 112 Display control unit 102 Learning device 122 Learning unit 103 Manipulation detection server 131 Image acquisition unit 132 Detection unit 104 OCR server 11 Target manuscript 21 Target image (scanned image)
32 Detection result data

Claims (16)

A generation means for generating a learning model by executing machine learning processing based on a falsified image, an image before falsification, and an image showing the difference between the falsified image and the image before falsification. When,
Input means for inputting image data and
An image processing system comprising the estimation means for estimating whether or not the image data input by the input means includes a falsified image by using the learning model generated by the generation means. .. Has more display means,
Since it is estimated by the estimation means that the image data input by the input means includes a falsified image, the display means emphasizes and displays the falsified part of the falsified image. The image processing system according to claim 1, wherein the image processing system is characterized. The image processing system according to claim 2, wherein the display means displays an image indicated by image data input by the input means together with an image in which the tampered portion is emphasized. Reception means and
The first determining means for determining that the designated pixel is a falsified pixel by receiving the designation of the pixel by the user in the falsified image displayed by the display means.
Further, the receiving means further comprises a second determining means for determining that the designated pixel is not a falsified pixel by accepting a pixel designation by the user in the falsified image displayed by the display means. The image processing system according to claim 2 or 3, wherein the image processing system has. The display means is characterized in that it displays information indicating that falsification has not been detected by presuming that the image data input by the input means does not include a falsified image. The image processing system according to any one of claims 2 to 4. The learning means includes a falsified image indicated by the image data input by the input means, an image before being falsified indicated by the image data input by the input means, and a falsified image before being falsified. The image processing system according to any one of claims 1 to 5, wherein a learning model is generated by executing a machine learning process based on an image showing a difference between images. A learning model updated by the machine learning process is specified based on the falsified image indicated by the image data input by the input means and the image before falsification indicated by the image data input by the input means. The image processing system according to claim 6, further comprising a third determination means for determining based on the identification information. Further having a storage means for storing the learning model and the identification information in association with each other,
The third determination means determines the learning model corresponding to the designated identification information as a learning model updated by performing the machine learning process based on the image before falsification input by the input means. The image processing system according to claim 7, wherein the image processing system is characterized by the above. A determination means for determining whether the image indicated by the image data input by the input means is a falsified image, or whether the image indicated by the image data input by the input means is an image before falsification.
The image data determined by the determination means to be an image in which the image indicated by the image data has been tampered with, and the image data determined by the determination means to be an image before the image indicated by the image data has been tampered with. The image processing system according to any one of claims 1 to 8, further comprising a storage means for storing in association with each other. Based on the identifier included in the image data input by the input means, the storage means determines that the image indicated by the image data is a falsified image, and the determination means determines that the image data is a falsified image. The image processing system according to claim 9, wherein the image indicated by the image data is associated with the image data determined to be an image before being tampered with and stored. The image processing system according to claim 10, wherein the identifier is a two-dimensional code. The image processing system is composed of at least an image processing device, a learning device, and an estimation device.
The image processing device has the input means.
The learning device has the learning means and
The image processing system according to any one of claims 1 to 11, wherein the estimation device includes an estimation means. The image processing device is
Further, it has a transmission means for transmitting the image data input by the input means to the learning device.
The learning device is
It further has a receiving means for receiving the image data transmitted from the image processing apparatus by the transmitting means.
The image processing system according to claim 12, wherein the learning means executes the machine learning process based on the image data received by the receiving means. The image processing system includes at least the image processing device, the learning device, the estimation device, and a character recognition device.
The character recognition device has a character recognition means for executing character recognition processing.
The estimation means of the estimation device is based on the image data input by the image processing device, the learning model generated by the learning device, and the result of the character recognition processing output by the character recognition device. The image processing system according to claim 12 or 13, wherein it estimates whether or not the image data input by the input means includes a falsified image. It is an image processing method
A generation process that generates a learning model by executing machine learning processing based on a falsified image, an image before falsification, and an image showing the difference between the falsified image and the image before falsification. When,
Input process for inputting image data and
An image processing method characterized by having an estimation step of estimating whether or not a falsified image is included in the image data input in the input step using the learning model generated in the generation step. .. A program for executing the image processing method according to claim 15 on a computer.

Images