JP2023081269A - Image forming device, method for controlling image forming device, and program - Google Patents

Abstract

To provide an image forming device capable of highly accurately estimating existence of a graph and a table without reference to the types of the graph and the table if the graph and the table exist in a document, a method for controlling the image forming device, and a program.SOLUTION: An image forming device (image processing device) 101 is a device for processing an image. The image forming device 101 comprises a scanner unit (reading means) 10 for reading a document, and an estimation processing unit (estimation processing means) 405 for estimating a chart area in the document read by the scanner unit 10 by using a learned model in which machine learning was performed which estimates a chart area of at least a graph region containing graphs in the document or a table area containing tables.SELECTED DRAWING: Figure 6

Description

The present invention relates to an image forming apparatus, an image forming apparatus control method, and a program.

2. Description of the Related Art As an image forming apparatus for forming an image on a recording material such as paper, there is known a multifunction machine having a printer function, a scan function, a facsimile function, a copy function, and the like. A multifunction device can read a color-printed document (referred to as a "primary document" here) using a scanning function. When outputting the read primary document using the printer function, the document (herein referred to as a "secondary document") may be output by monochrome printing in order to reduce printing costs. For example, if the primary manuscript contains two bar graphs, a red bar graph and a blue bar graph, in the secondary manuscript, these two bar graphs differ only in shades, and their identifiability is not achieved by color printing. is inferior to the case of In order to address such a problem, the image forming apparatus disclosed in Japanese Patent Application Laid-Open No. 2002-200313 can apply a pattern (hatching) that is easy to identify even in monochrome printing to each bar graph. Further, in the image forming apparatus disclosed in Patent Document 1, when a graph in a primary document is detected, if the shape of a colored (filled) area is a rectangle or a sector, the area is is detected as a graph.

JP 2016-165026 A

However, in the image forming apparatus disclosed in Japanese Patent Application Laid-Open No. 2002-200311, if a colored rectangular or fan-shaped area exists in the primary document, the entire area is erroneously detected as a graph. Also, if the primary document includes a graph that is not rectangular or fan-shaped, such as a line graph or a 3D graph, the graph cannot be detected.

INDUSTRIAL APPLICABILITY The present invention is an image forming apparatus capable of estimating the existence of a graph or a table with high accuracy regardless of the type of the graph or the table when the document contains the graph or the table, and control of the image forming apparatus. It aims at providing a method and a program.

To achieve the above object, an image forming apparatus of the present invention is an image forming apparatus for processing an image, comprising reading means for reading a document, a graph area including a graph in the document, and a table area including a table. estimation processing means for estimating the graphic area in the document read by the reading means using a learned model subjected to machine learning for estimating the graphic area of at least one of characterized by

According to the present invention, when a document contains graphs and tables, the existence of the graphs and tables can be estimated with high accuracy regardless of the types of graphs and tables.

1 is a diagram showing the overall configuration of an image forming system; FIG. 2 is a block diagram showing the hardware configuration of the image forming apparatus; FIG. It is a block diagram which shows the hardware constitutions of a machine-learning server. 2 is a block diagram showing the software configuration of the image forming system; FIG. FIG. 4 is a conceptual diagram showing an input/output structure in a learning model (graph relation); FIG. 4 is a conceptual diagram showing an input/output structure in a learning model (table relationship); FIG. 4 is a conceptual diagram showing the input/output structure in the learning model; FIG. 3 is a diagram for explaining operations (process flow) performed in the image forming system; FIG. 4 is a diagram showing a configuration example of a document read by a scanner unit and an estimation result (graph relation); 4A and 4B are diagrams showing a configuration example of a document read by a scanner unit and an estimation result (table relation); FIG. 7B is a table corresponding to FIG. 7A; FIG. 7B is a table corresponding to FIG. 7B; 4 is a flow chart showing the flow of a scan-related job; 4 is a flow chart showing the flow of processing by a machine learning unit and an estimation processing unit; FIG. 10 is a diagram showing a display example on a UI display unit;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in the embodiments.

<Overall Configuration of Image Forming System>
FIG. 1 is a diagram showing the overall configuration of an image forming system. As shown in FIG. 1 , an image forming system (image processing system) 1000 has an image forming apparatus (image processing apparatus) 101 , a machine learning server 102 , a general-purpose computer 103 and a data server 105 . The image forming apparatus 101, the machine learning server 102, the general-purpose computer 103, and the data server 105 are communicably connected to each other via a network 104 such as a wired LAN. The image forming apparatus 101 is composed of, for example, a printer, a multifunction machine, a FAX, etc., and can form an image on a recording material such as paper. The image forming apparatus 101 is equipped with an AI function, and can receive a learned model from the machine learning server 102 at any time to implement the AI function. The machine learning server 102 mainly receives from the data server 105 learning data necessary for machine learning of a trained model for realizing the AI function. Note that the machine learning server 102 can also receive learning data from devices other than the data server 105 . The data server 105 collects learning data used for machine learning by the machine learning server 102 from external devices such as the image forming apparatus 101 and the general-purpose computer 103 and transmits the learning data to the machine learning server 102 . The general-purpose computer 103 is, for example, a personal computer, and performs transmission of print data to the image forming apparatus 101 and the like.

In the image forming system 1000, the data server 105 collects the data of the document read by the image forming apparatus 101, and the machine learning server 102 performs machine learning on the collected data to generate a learning model. This learning model is a learning model for estimating the presence/absence of graphs and tables in the manuscript loaded from the machine learning server 102 . The image forming apparatus 101 has an AI function that utilizes learning models. With such an image forming system 1000, it is possible to determine whether or not a document read by the image forming apparatus 101 contains a graph or a table, and prompt the user to perform subsequent processing (for example, change document data). It has become.

<Hardware Configuration of Image Forming Apparatus (Controller Configuration of Image Forming Apparatus)>
FIG. 2 is a block diagram showing the hardware configuration of the image forming apparatus. As shown in FIG. 2 , the image forming apparatus 101 has an operation section (operation means) 140 , a scanner section (reading means) 10 and a printer section (printing means) 20 . The operation unit 140 has, for example, a liquid crystal display equipped with a multi-touch sensor, etc., and receives various operations from the user who uses the image forming apparatus 101 . The scanner unit 10 operates according to an operation from the user via the operation unit 140, and can read an image (image information) formed on a document (reading process). The scanner unit 10 includes a CPU that controls the scanner unit 10, an illumination lamp that irradiates light when reading a document, a scanning mirror that reflects light, and the like. The printer section 20 can print an image on paper. As a result, a color-printed document or a monochrome-printed document can be obtained. The printer section 20 includes a CPU for controlling the printer section 20, a photosensitive drum for forming an image on paper, a fixing device for fixing an image on the paper, and the like.

The image forming apparatus 101 also has a controller 1200 . The controller 1200 is communicably connected to the scanner unit 10, the printer unit 20, the network 104, the wireless LAN 106, and the public line (WAN) 3001, and controls the operation of the image forming apparatus 101 in an integrated manner. The controller 1200 has a raster image processor (RIP) 1260 , scanner image processing section 1280 , printer image processing section 1290 , image rotation section 1230 , image compression section 1240 , device I/F 1220 and image bus 2008 . A raster image processor (RIP) 1260 develops PDL code included in a print job received from the general-purpose computer 103 via the network 104 into a bitmap image. A scanner image processing unit 1280 corrects, processes, and edits image data input from the scanner unit 10 . A printer image processing unit 1290 performs correction, resolution conversion, and the like on image data output (printed) by the printer unit 20 . The image rotation unit 1230 rotates image data. The image compression unit 1240 performs JPEG compression/decompression processing for multilevel image data, and JBIG, MMR, or MH compression/decompression processing for binary image data. A device I/F 1220 connects the scanner unit 10 and the printer unit 20 to the controller 1200 and performs synchronous/asynchronous conversion of image data. An image bus 2008 connects these to each other to transfer image data at high speed.

Controller 1200 also has CPU 1201 , RAM 1202 , operation unit I/F 1206 , network unit 1210 , modem unit 1211 , and wireless communication I/F 1270 . A CPU 1201 is a control unit that controls the image forming apparatus 101 in an integrated manner. A RAM 1202 is a system work memory for the operation of the CPU 1201 and also functions as an image memory for temporarily storing image data. Operation unit I/F 1206 outputs image data to be displayed on operation unit 140 to operation unit 140 . Further, the operation unit I/F 1206 has a role of transmitting information input by the user from the operation unit 140 to the CPU 1201 . The network unit 1210 is connected to the network 104 and communicates (transmits and receives) with the general-purpose computer 103 and other computer terminals (not shown) on the network 104 . A modem unit 1211 is connected to the public line 3001 and performs data communication (transmission/reception) with an external facsimile machine (not shown). A wireless communication I/F 1270 connects with an external terminal via the wireless LAN 106 .

The controller 1200 also has a ROM 1203 , HDD (hard disk drive) 1204 , system bus 1207 , internal communication I/F 1208 , image bus 1212 and ImageBus I/F 1205 . The ROM 1203 stores programs. As this program, for example, there is a program for causing the CPU 1201 (computer) to execute the operation of each unit and means of the image forming apparatus 101 (control method of the image forming apparatus). A hard disk drive (HDD) 1204 stores system software, image data, software counter values, and the like. The controller 1200 records and manages a history of job execution such as user name, number of copies, color printing, output attribute information, etc., in the HDD 1204 or RAM 1202 as job log information. An internal communication I/F 1208 communicates with the scanner section 10 and the printer section 20 . The ImageBus I/F 1205 connects the system bus 1207 and the image bus 1212 and functions as a bus bridge for converting data structures.

Also, the controller 1200 has a GPU 1291 . The GPU 1291 can perform efficient operations by processing more data in parallel. Therefore, when learning is performed multiple times using a learning model such as deep learning, it is effective to perform processing using the GPU 1291 . In this embodiment, a GPU 1291 is used in addition to the CPU 1201 for processing by the machine learning unit 414, which will be described later. Specifically, when a learning program including a learning model is executed, the CPU 1201 and the GPU 1291 cooperate to perform calculations for learning. Note that the processing of the machine learning unit 414 may be performed only by the CPU 1201 or the GPU 1291 . Also, the GPU 1291 may be used for the processing of the estimation processing unit (estimation processing means) 405 described later, similarly to the machine learning unit 414 .

<Hardware configuration of machine learning server>
FIG. 3 is a block diagram showing the hardware configuration of the machine learning server. As shown in FIG. 3, the machine learning server 102 has a CPU 1301, a RAM 1302, a ROM 1303, a HDD (Hard Disk Drive) 1304, a network section 1310, an IO section 1305, and a GPU 1306. Also, they are connected to each other via a system bus 1207 . The CPU 1301 provides various functions by reading programs such as an OS (Operating System) and application software from the HDD 1304 and executing them. A RAM 1302 is a system work memory when the CPU 1301 executes programs. The ROM 1303 stores a BIOS (Basic Input Output System), programs for starting the OS, and setting files. The HDD 1304 is a hard disk drive and stores system software and the like. A network unit 1310 is connected to the network 104 and communicates (transmits and receives) with external devices such as the image forming apparatus 101 . The IO unit 1305 constitutes an operation unit (not shown) together with a liquid crystal display input/output device having a multi-touch sensor or the like. The IO unit 1305 functions as an interface for inputting/outputting information to/from the operation unit. Predetermined information is drawn on the operation unit with a predetermined resolution, number of colors, etc., based on screen information instructed by the program. For example, a GUI (Graphical User Interface) screen is formed in the operation unit, and various windows and data required for operation are displayed.

The GPU 1306 can perform efficient operations by processing data in parallel. Therefore, when learning is performed multiple times using a learning model such as deep learning, it is effective to perform processing on the GPU 1306 . In this embodiment, a GPU 1306 is used in addition to the CPU 1301 for processing by the machine learning unit 414 . Specifically, when a learning program including a learning model is executed, the CPU 1301 and the GPU 1306 cooperate to perform calculations for learning. Note that the processing of the machine learning unit 414 may be performed only by the CPU 1301 or the GPU 1306 . Also, the GPU 1306 may be used for the processing of the estimation processing unit 405 as well as the machine learning unit 414 . Here, proper use of the GPU 1291 of the image forming apparatus 101 will be described. In the image forming system 1000, the computational resources of the GPU are effectively utilized according to the communication via the network 104, the processing load of the GPU, the power saving mode of the image forming apparatus 101, and the like. For example, when the image forming apparatus 101 shifts to the power saving mode, the GPU 1301 on the machine learning server 102 side is preferentially (actively) utilized.

<Software Configuration of Image Forming System>
FIG. 4 is a block diagram showing the software configuration of the image forming system. A software configuration 400 shown in FIG. 4 is realized by using hardware (see FIG. 2) configuring the image forming apparatus 101 and programs. A program for realizing the software configuration 400 is stored in the storage for each component, read out to the RAM, and executed by the CPU. For example, in the image forming apparatus 101 , programs are stored in the HDD 1204 , read out to the RAM 1202 and executed by the CPU 1201 . The same applies to the machine learning server 102 and data server 105 . The software configuration 400 enables the image forming system 1000 to implement a function of estimating the presence/absence of graphs and tables using learning data read by the image forming apparatus 101 .

As shown in FIG. 4, the software of the image forming apparatus 101 includes a UI display unit (display means) 401, a data storage unit (storage means) 402, a JOB control unit 403, an image reading unit 404, and an estimation processing unit 405. be. The data storage unit 402 has a functional role of causing the RAM 1202 and HDD 1204 of the image forming apparatus 101 to record various data such as image data, learning data, and learning models. The JOB control unit 403 has a functional role of centrally executing basic functions of the image forming apparatus 101 such as copying, faxing, and printing based on user instructions via the operation unit 140 of the image forming apparatus 101 . . In addition to executing the basic functions, the JOB control unit 403 also plays a central role in instructing, transmitting and receiving data between other software components associated with the execution of the basic functions. The UI display unit 401 has a functional role of displaying a notification screen for notifying the user of a message on the screen of the operation unit 140 . Examples of the message include reception of operation settings from the user, provision of the operation reception screen, estimation results by the estimation processing unit 405, and the like. The image reading unit 404 has a functional role of causing the scanner unit 10 to optically read a document when performing control to execute copying or scanning based on instructions from the JOB control unit 403 . The estimation processing unit 405 is executed by the CPU 1201 and GPU 1291 of the image forming apparatus 101 . The estimation processing unit 405 has a functional role of performing estimation processing, classification processing, and the like for realizing the AI function on data input/output by the image forming apparatus 101 . This estimation processing is performed based on instructions from the JOB control unit 403 . Also, the result of the estimation processing unit 405 is transmitted to the JOB control unit 403 and displayed on the UI display unit 401 as a message. Thereby, the user can grasp the result of the estimation processing unit 405 .

The software of the data server 105 is composed of a data collecting/providing unit 410 and a data storage unit 412 . The data collecting/providing unit 410 has a functional role of collecting and providing learning data for machine learning in the machine learning server 102 . Specifically, in the image forming system 1000 , the image forming apparatus 101 transmits learning data including operation information that the image forming apparatus 101 has been operated. The data collecting/providing unit 410 receives learning data from the image forming apparatus 101 and provides the learning data to the machine learning server 102 . Also, the collection destination may be an image forming apparatus other than the image forming apparatus 101, a general-purpose computer 103, or a data server other than the data server 105. FIG. This makes it possible to collect the data necessary for performing the desired machine learning. The data storage unit 412 has a functional role of recording and managing learning data collected by the data collecting/providing unit 410 .

The software of the machine learning server 102 is composed of a learning data generation unit 413 , a machine learning unit 414 and a data storage unit 415 . The learning data generator 413 receives various data including learning data from the data server 105 . Then, the learning data generation unit 413 processes this data into a form that provides effective learning results, such as removing unnecessary data that becomes noise in machine learning. Thus, the learning data generator 413 has a functional role of optimizing the learning data. The learning data generator 413 is executed by the CPU 1301 of the machine learning server 102 . The data storage unit 415 temporarily records the data received from the data server 105, the learning data generated by the learning data generation unit 413, and the trained model in the machine learning unit 414 in the RAM 1302 and HDD 1304 of the machine learning server 102. . The machine learning unit 414 uses the learning data generated by the learning data generation unit 413 as input data, and the GPU 1306 and CPU 1301, which are hardware resources of the machine learning server 102, and the learning model W shown in FIG. to perform machine learning.

<Learning model>
5A, 5B, and 5C are conceptual diagrams showing input/output structures in the learning model, respectively. As described above, the document is read by the scanner unit 10 . Then, the learning model W shown in FIG. 5A is adjusted through a learning process described later to become a learned model, and machine learning for estimating the graphic area in the document (document data A) read by the scanner unit 10 is performed. done. The chart area includes at least one of a chart area containing a graph and a table area containing a table. Also, the learning model W is, for example, a learning model using a neural network whose parameters are adjusted by the error backpropagation method or the like. As a result, the learning model W can perform deep learning for generating various parameters such as feature amounts and weights (connection weighting coefficients) for learning. Machine learning is not limited to deep learning, and may be machine learning using any machine learning algorithm such as support vector machine, logistics regression, decision tree, nearest neighbor method, naive Bayes method, etc. . Also, the learning model W may be a model using something other than a neural network.

The learning model W (learned model) uses, as input data, a manuscript to be estimated for a graph area or a table area, that is, information about a graph or a table in the manuscript data A. FIG. For example, in this embodiment, as shown in FIG. 5A, the input data includes data element X0, axis X1, figure number X2, title X3, legend X4, scale line X5, and text X6. . As shown in FIG. 5B, the table includes data element X10, table number X11, cell X12, ruled line X13, row X14, column X15, and text X16.

Data element X0 is the graph itself, and in this embodiment the graph area contains multiple graphs. For example, in FIG. 5A, there are three bar graphs for February, three bar graphs for March, and three bar graphs for April. Also, in the manuscript (manuscript data A), at least the graph is printed in color. In this case, the three bar graphs for each month are colored differently. The bar graphs positioned on the left side of each month, the bar graphs positioned in the center, and the bar graphs positioned on the right side of each month are colored in the same color. For example, the bar graphs positioned on the left side of each month are colored blue, the bar graphs positioned in the center are colored red, and the bar graphs positioned on the right side are colored gray. Axis X1 is a vertical axis or a horizontal axis. Axis X1 may also include a label indicating what each axis represents, a unit for each axis, and the like. The figure number X2 is the serial number of the graph and the name of the graph. Title X3 is the name of the graph. The legend X4 is an item explaining what each graph represents. In FIG. 5A, the legend X4 explains what each of the three bar graphs for each month represents. It represents that it is "c". A scale line X5 is a scale line indicating the amount of the graph. Text X6 is a sentence that includes the noun "graph".

The data element X10 is the table itself, and the table area includes one table in this embodiment, but is not limited to this and may include multiple tables. Also, the table is displayed in color printing in the same manner as the graph. In this case, a plurality of cells X12 are colored. Cell X12 is an element representing a data element of the table. Each cell X12 of "e" in February, "h" in March and "h" in April, and each cell X12 of "g" in February, "f" in March and "e" in April are colored differently from each other. For example, each cell X12 of "e" in February, "h" in March, and "h" in April is colored orange, and "g" in February, "f" in March, and "h" in April. Each cell X12 of "e" is colored green. In FIG. 5B, each cell X12 of "e" in February, "h" in March, and "h" in April indicates the minimum number of shipments for each month, and "g" in February, 3 The maximum number of shipments is shown in each cell X12 of "f" for month and "e" for April. The table number X11 is the serial number of the table and the name of the table. A ruled line X13 indicates a break that constitutes each cell X12 of the table. Row X14 and column X15 indicate the data elements of the table arranged vertically and horizontally. Text X16 is a sentence that includes the noun "table".

Also, the learning model W (learned model) uses the presence/absence of graph areas and table areas as figure/table areas as output data. For example, in this embodiment, the output data is "output data Y1" when there is a graph or table, and "output data Y2" when there is no graph or table. The learning model W having such a configuration enables more accurate estimation of the graph area and the table area, that is, the estimation accuracy rate of the graph area and the table area is improved. Using the learning model W (learned model), the estimation processing unit 405 of the image forming apparatus 101 can estimate the graph area and the table area in the document read by the scanner unit 10 (estimation processing step ).

In the structure shown in FIG. 5C, the learning model W is prepared with a large number of learning data sets (sets) of "input data with known correct values" and "correct values" (see B1). This learning model W may have an error detector and an updater. The error detection unit calculates the output data Y (see B3) output from the output layer according to the input data X (see B2) input to the input layer, and outputs the output data Y (see B4). do. Then, the error detection section obtains the error between the output data Y and the teacher data T. FIG. The error detector uses a loss function to calculate a loss L representing the error between the output data Y and the teacher data T (see B5). Based on the loss L obtained by the error detection unit, the update unit updates the weighting coefficients for coupling between nodes of the neural network (learning model W) so as to reduce the loss (see B6). This updating unit updates the connection weighting coefficients and the like using, for example, the error backpropagation method. The error backpropagation method is a method of adjusting connection weighting coefficients and the like between nodes of a neural network so as to reduce the above error.

As described above, the learning model W is prepared with a large number of learning data sets of "input data with known correct values" and "correct values". Then, in the learning model W, the weighting coefficients in the learning model W are adjusted so that the output data Y when inputting the input data X corresponding to this correct value approaches the correct value as much as possible. As a result, a learning model W with a high accuracy of correct answer is obtained. The process of obtaining such a learning model W is called a "learning process", and the learning model adjusted through the learning process is called a "learned model". Also, the training data to be prepared, that is, the set of "input data with known correct values" and "correct values" can be, for example, as follows.

Input data X for which correct values are known: Information about graphs and tables in manuscript data in which graphs and tables exist. This information is not particularly limited, and includes, for example, the aforementioned axis X1, figure number X2, title X3, legend X4, scale line X5, text X6, and the like.
Expected value (T): A (with graphs and tables) = 1, B (without graphs and tables) = 0
The expected value (T) is the value of the output data Y indicating the "correct value" when "the input data whose correct value is known" is input. The expected value (T) is "output data Y1" when there is a graph or table, and "output data Y2" when there is no graph or table. A loss L is obtained from the output data Y calculated by inputting the input data X of each teacher data T and the expected value (T) according to a predetermined loss function. The loss function in this embodiment is assumed to be "loss L=1-estimated probability of output data Y". The weighting between each layer of the learning model W is adjusted so that the loss L approaches zero. Then, the adjusted learning model W is used as a learned model, and the learned model is implemented in the machine learning unit 414 .

<Operations (Processing Flow) Performed by the Image Forming System>
FIG. 6 is a diagram for explaining the operation (process flow) performed in the image forming system. As shown in FIG. 6, in the image forming system 1000, operations I to V are executed in order.
I: When the image forming apparatus 101 receives a scanning operation for a document from the user via the operation unit 140, the image forming apparatus 101 transmits scanned data to the machine learning server 102 after the scanning operation. Along with this transmission, the image forming apparatus 101 also requests estimation of a graph area and a table area (graphic area) for the scan data.
II: The machine learning server 102 that has been requested to estimate a graph area or a table area in operation I estimates the presence or absence of a graph area or table area in the scan data using a learning model. Note that in operation II, the image forming apparatus 101 may receive a learning model from the machine learning server 102 and use the learning model to estimate whether there is a graph area or a table area in the scan data.
III: The machine learning server 102 transmits to the image forming apparatus 101 the result of estimating the presence/absence of the graph area and table area obtained in operation II.
IV: When the image forming apparatus 101 prints the graphs and tables in the document having the estimation results transmitted from the machine learning server 102 in operation III in monochrome, the image forming apparatus 101 improves the identifiability (visibility) of the graphs and tables. A maintainable correction example (sample image) is displayed on the operation unit 140 . This makes it possible to propose correction examples to the user.
V: When the image forming apparatus 101 corrects the graph or table according to the correction example, it saves the data of the correction example.

By executing the operations I to V shown in FIG. 6 in order, the presence/absence of graphs and tables in the scan data is estimated. If the identifiability deteriorates, a correction example can be suggested to the user. Then, it becomes possible to correct the scan data based on the correction example.

<Example of composition of manuscript read by scanner unit (collection of training data)>
FIG. 7A is a diagram showing a configuration example of a document read by the scanner unit and an estimation result (graph relation). FIG. 7B is a diagram showing a configuration example of a document read by the scanner unit and an estimation result (table relationship); FIG. 8A is a table corresponding to FIG. 7A. FIG. 8B is a table corresponding to FIG. 7B.

The document shown in FIG. 7A(a) is a document in which a graph area including a rectangular or fan-shaped graph (a bar graph in FIG. 7A(a)) is formed as an image. As shown in FIGS. 7A(a-1) and 8A, in the manuscript shown in FIG. 7A(a), input data X includes data element X0, axis X1, title X3, legend X4, scale line X5, and text X6. Yes, but no figure number X2.

The document shown in FIG. 7A(b) is a document in which a graph area including a rectangular or fan-shaped graph (circular graph in FIG. 7A(b)) is formed as an image. As shown in FIGS. 7A(b-1) and 8A, in the document shown in FIG. 7A(b), input data X includes data element X0, figure number X2, title X3, legend X4, axis X1, scale There is no line X5 and text X6.

The document shown in FIG. 7A(c) is a document in which a graph area including a non-rectangular or fan-shaped graph (line graph in FIG. 7A(c)) is formed as an image. As shown in FIGS. 7A(c-1) and 8A, in the document shown in FIG. 7A(c), there are data element X0, axis X1, title X3 to scale line X5 as input data X, and figure numbers X2, There is no text X6.

The document shown in FIG. 7A(d) is a document in which a graph area including a non-rectangular or fan-shaped graph (a 3D line graph in FIG. 7A(d)) is formed as an image. As shown in FIGS. 7A(d-1) and 8A, in the document shown in FIG. 7A(d), the input data X includes the data elements X0 to the scale line X5 and does not include the text X6.

The document shown in FIG. 7A(e) is a document in which an area including a rectangular or fan-shaped figure that is not a graph is formed as an image. As shown in FIGS. 7A(e-1) and 8A, in the document shown in FIG. 7A(e), as the input data X, all of the data elements X0 to X6 are absent.

The document shown in FIG. 7A(f) is a document in which a graph area including a non-rectangular or fan-shaped graph (an annular graph in FIG. 7A(f)) is formed as an image. As shown in FIGS. 7A(f-1) and 8A, the manuscript shown in FIG. 7A(f) has legend X4 as input data X, and does not have data elements X0 to title X3, scale lines X5, and text X6. .

The document shown in FIG. 7B(g) is a document in which a table area including a table (in FIG. 7B(g), a table including ruled lines in all cells) is formed as an image. As shown in FIGS. 7B(g-1) and 8A, in the document shown in FIG. 7B(g), the input data X includes data element X10, table number X11, cell X12, ruled line X13, row X14, and column X15. Yes, but no text X16.

The document shown in FIG. 7B(h) is a document in which a table area including a table (in FIG. 7B(h), a table without some of the cell ruled lines) is formed as an image. As shown in FIG. 7B(h-1) and FIG. 8A, in the document shown in FIG. 7B(h), input data X includes data element X10, cell X12, ruled line X13, row X14, and column X15. There is no X11 and text X16.

For example, an estimation process when the learning model is in an unlearned state will be described. Here, the "unlearned state" includes not only an initial state in which learning is not performed at all, but also a state in which learning is performed but is insufficient. In this case, the learning model is in a state where only bar graphs, pie charts, and line graphs have been learned as the data element X0.

In the case of the manuscript shown in FIG. 7A(a), the data element X0 has already been learned, and the other input data X are sufficiently complete to be able to estimate that "there is a graph area". Therefore, it can be estimated that "there is a graph area" for the document shown in FIG. 7A(a). As shown in FIG. 8A, this estimation result becomes teacher data T, that is, "T1=1, T2=0" as the expected value (T).

In the case of the manuscript shown in FIG. 7A(b), the data element X0 has been learned, but the other input data X is insufficient to estimate the graph area. Therefore, the manuscript shown in FIG. 7A(b) is estimated to have "no graph area". Therefore, in such a case, the user corrects "there is a graph area" and updates the learned model. As a result, it can be accurately estimated that "there is a graph area" for the document shown in FIG. 7A(b). As shown in FIG. 8A, this estimation result becomes teacher data T of "T1=1, T2=0".

In the case of the document shown in FIG. 7A(c), it is estimated that "there is a graph area" as in the case of the document shown in FIG. 7A(a). As shown in FIG. 8A, this estimation result becomes teacher data T of "T1=1, T2=0".

In the case of the manuscript shown in FIG. 7A(d), the data element X0 has not been learned at all, but the other input data X are sufficiently prepared so that it can be estimated that "there is a graph area". Therefore, it can be estimated that "there is a graph area" for the document shown in FIG. 7A(d). As shown in FIG. 8A, this estimation result becomes teacher data T of "T1=1, T2=0". As described above, even unlearned graphs can be correctly estimated for a manuscript having a graph area.

In the case of the manuscript shown in FIG. 7A(e), although the figure in the manuscript has a shape similar to the data element X0, the input data X does not have all of the axis X1 to the text X6. Therefore, the manuscript shown in FIG. 7A(e) is presumed to have "no graph area". As shown in FIG. 8A, this estimation result becomes teacher data T, that is, "T1=0, T2=1" as the expected value (T). Further, in the image forming apparatus described in Patent Document 1, the original shown in FIG.

In the case of the document shown in FIG. 7A(f), the data element X0 has not been learned at all, and the other input data X has only the legend X4. Therefore, the manuscript shown in FIG. 7A(f) is estimated to have "no graph area". In such a case, as in the case of the manuscript shown in FIG. 7A(b), the user corrects "there is a graph area" and updates the learned model. As a result, it can be accurately estimated that "there is a graph area" for the document shown in FIG. 7A(f). As shown in FIG. 8A, this estimation result becomes teacher data T of "T1=1, T2=0".

In the case of the manuscript shown in FIG. 7B(g), the data element X10 has already been learned, and the other input data X are sufficiently prepared to be estimated as "with table area". As shown in FIG. 8A, this estimation result becomes teacher data T, that is, "T1=1, T2=0" as the expected value (T).

In the case of the document shown in FIG. 7B(h), the data element X10 has already been learned, but among the other input data X, for example, the table number X11 and the ruled line X13 are insufficient. It is sufficient enough to be estimated. Therefore, it can be estimated that the document shown in FIG. 7B(h) has "a front area". In this way, it is possible to correctly estimate the table area even for a table whose elements are slightly different from the already learned table.

As described above, even for an unlearned data element X0, if enough other input data X exist, the graph area or table area can be accurately estimated. Moreover, even with the data element X0 that has already been learned, if the other input data X is insufficient, there is a risk that the graph area or the table area will be erroneously estimated. However, in the image forming apparatus 101, by correcting the erroneous estimation result, the learning model can be updated to improve the accuracy of estimating the presence/absence of graph regions and table regions. As described above, according to the image forming apparatus 101, when a document contains a graph or a table, the existence of the graph or table can be estimated with high accuracy regardless of the type of the graph or table.

<Flow of scan-related jobs>
FIG. 9 is a flow chart showing the flow of a scan-related job. Note that the “scan-related jobs” refer to all jobs using a scanner, such as copying and scanning BOX.

In step S1101, the job control unit 403 determines whether or not the operation unit 140 has received a start of a scan-related job from the user. As a result of the determination in step S1101, if the JOB control unit 403 determines that the operation unit 140 has accepted the start of the job, the process advances to step S1102. On the other hand, if the JOB control unit 403 determines in step S1101 that the operation unit 140 has not accepted the start of the job, it waits until the operation unit 140 accepts the start of the job.

In step S1102, the job control unit 403 activates the scanner unit 10 (image reading unit 404) to read a document (hereinafter referred to as "primary document"). The primary document is read by the scanner section 10 while being placed on an ADF (Automatic Document Feeder) or a platen glass (neither of which is shown).

In step S1103, the job control unit 403 determines whether or not the scanner unit 10 has finished reading all the originals. As a result of the determination in step S1103, if the JOB control unit 403 determines that reading of the document is completed, the process advances to step S1104. On the other hand, if the JOB control unit 403 determines in step S1103 that reading of the document has not been completed, the process is executed until reading of all the documents is completed.

In step S1104, the job control unit 403 saves the primary document data read in step S1103 in the data storage unit 402. FIG.

In step S1105, the JOB control unit 403 activates the machine learning unit 414 of the machine learning server 102 to perform learning processing. Step S1105 is a subroutine, which will be described later with reference to the learning phase flow chart shown in FIG.

In step S1106, the JOB control unit 403 activates the estimation processing unit 405 to perform estimation processing for estimating the graph area and the table area. Step S1106 is a subroutine, which will be described later with reference to the estimation phase flowchart shown in FIG.

In step S1107, the job control unit 403 executes post-reading processing, and the processing ends. Post-reading processing includes, for example, printing a document reflecting the result of the estimation process (hereinafter, this document is referred to as a “secondary document”), storing data of the secondary document, and the like. The secondary document is printed by the printer section 20. FIG. Data of the secondary original is saved in the data storage unit 402 .

<Processing Flow of Machine Learning Unit and Estimation Processing Unit>
FIG. 10 is a flow chart showing the processing flow of the machine learning unit and the estimation processing unit. In FIG. 10, steps S901 to S906 are the processing of the machine learning unit performed in the learning phase, and steps S907 to S912 are the processing of the estimation processing unit performed in the estimation phase. The machine learning unit 414 confirms whether or not the learning data has been updated at regular intervals. Then, the machine learning unit 414 starts the learning phase when it determines that the learning data has been updated.

In step S<b>901 , the machine learning unit 414 receives learning data via the learning data generation unit 413 . The learning data is the manuscript data A described above.

In step S902, the machine learning unit 414 inputs the input data X of the document data A received in step S901 to the learning model W, and the learning model W performs machine learning.

In step S903, the machine learning unit 414 determines whether or not machine learning has ended. As a result of the determination in step S903, when the machine learning unit 414 determines that the machine learning has ended, the process proceeds to step S904. On the other hand, when the machine learning unit 414 determines that the machine learning has not ended as a result of the determination in step S903, the execution of step S902 is continued.

In step S904, the machine learning unit 414 notifies the estimation processing unit 405 of updating the learned model.

In step S<b>905 , the machine learning unit 414 determines whether or not the estimation processing unit 405 has requested transmission of the trained model. As a result of the determination in step S905, if the machine learning unit 414 determines that there is a transmission request for the trained model, the process proceeds to step S906. On the other hand, as a result of the determination in step S905, when the machine learning unit 414 determines that there is no transmission request for the trained model, the process returns to step S905 and waits for a transmission request from the estimation processing unit 405.

In step S<b>906 , the machine learning unit 414 transmits the learned model to the estimation processing unit 405 .

In step S907, the estimation processing unit 405 determines whether the job control unit 403 has received a scan-related job from the user. This scan-related job is to read the primary document. As an example, the primary document here is the document shown in FIG. 7A(a). As a result of the determination in step S907, if the estimation processing unit 405 determines that the job control unit 403 has received a scan-related job, the process advances to step S908. On the other hand, as a result of the determination in step S907, if the estimation processing unit 405 determines that the JOB control unit 403 has not accepted a scan-related job, it waits until the JOB control unit 403 accepts a scan-related job. .

In step S908, the estimation processing unit 405 requests the machine learning unit 414 to transmit the learned model via the JOB control unit 403, and receives the learned model.

In step S909, the job control unit 403 determines whether or not the primary document for which it was determined in step S907 that there was a scan-related job was color-scanned. As a result of the determination in step S909, if the job control unit 403 determines that the primary document has been color-scanned, the process proceeds to step S910. On the other hand, if the job control unit 403 determines in step S909 that the primary document has not been color-scanned, the process ends.

In step S910, the estimation processing unit 405 estimates a graph area and a table area, that is, a chart area in the color-scanned primary document using a learned model. If it is estimated in step S910 that there is no graph area or table area in the primary document, the process ends.

In step S911, the estimation processing unit 405 extracts each bar graph (data element X0) in the graph area and the data element X10 in the table area by known image processing. Then, the estimation processing unit 405 determines whether or not each bar graph in the graph area has the same filling pattern, that is, whether or not the bar graphs are colored. The estimation processing unit 405 also determines whether or not the data element X10 in the table region has the same filling pattern, that is, whether or not it is colored in a single color. As a result of the determination in step S911, if the estimation processing unit 405 determines that the fill patterns are the same, the process proceeds to step S912. On the other hand, if the estimation processing unit 405 determines that the fill patterns are not the same as a result of the determination in step S911, the process ends. If the fill patterns are not the same, even if the primary manuscript is printed in monochrome to obtain a secondary manuscript, the identifiability (visibility) of each bar graph in the secondary manuscript is not as good as that of color printing. The identifiability (visibility) between the bar graphs in the primary manuscript is maintained at approximately the same level.

The estimation processing unit 405 can change the visibility of each bar graph and table. The change in the visibility of the graphs and tables includes at least one of color change, shading change, and hatching change, and it is preferable to combine them. In addition, the estimation processing unit 405 can vary the visibility for each cell of the bar graph or table. For example, as shown in FIG. 7A (a-2), the bar graphs positioned on the left side of each month have dots, and the bar graphs positioned on the right side have dots with a higher density than the bar graphs positioned on the left side. , and the middle-located bars are dotted with intermediate densities. As a result, even if the primary manuscript is printed in black-and-white to obtain a secondary manuscript when the fill pattern is the same, the identifiability of each bar graph in the secondary manuscript will not be as high as in the color-printed primary manuscript. , the distinguishability between each bar graph is maintained almost the same. Note that, as shown in FIG. 7A(b-2), in the pie chart, dots having different densities are applied to each fan-shaped portion forming the pie chart. As shown in FIG. 7A(c-2), each line graph is represented by a different line. The line is not particularly limited, and for example, a solid line, a dashed line, a one-dot chain line, a two-dot chain line, or the like can be used. As shown in FIG. 7A(d-2), in the 3D line graph, dots with different densities are applied to each 3D line graph. As shown in FIG. 7A (f-2), in the circular ring graph, dots having different densities are applied to the circular arc-shaped portions forming the circular ring graph.

Then, in step S912, the estimation processing unit 405 causes the UI display unit 401 (operation unit 140), via the JOB control unit 403, to display an image of the chart area in which the identifiability (visibility) of the graph or table has been changed. Sample images can be displayed as correction candidates (see FIG. 11). The UI display unit 401 can also display the graph area and the table area in the state (scan data of the primary document) before the visibility of the graph is changed, side by side with the correction candidate (sample image). FIG. 11 is a diagram showing a display example on the UI display unit. As shown in FIG. 11, the UI display unit 401 displays a graph area of the primary document (scan data) arranged on the left side and a graph area after identification change (correction candidate) arranged on the right side. be done. This allows the user to compare (look and see) both graph areas and determine whether the bar graphs of the correction candidates are identifiable.

Further, as shown in FIG. 11, the UI display unit 401 displays a button for "correct to correction candidate", a button for "not correct", and a button for "correct manually". may When the user operates the button for “correct to correction candidate”, the UI display unit 401 corrects the graph area of the primary manuscript read by the scanner unit 10 . Make a decision to replace the candidate. Then, when this determination is executed, the printer unit 20 prints the secondary document on which the image including the correction candidate is formed by monochrome printing. As a result, even in the graph of the secondary manuscript printed in monochrome, the same level of distinguishability as in the graph of the primary manuscript printed in color is maintained. Further, the data storage unit 402 can store the data of the secondary document on which the image including the correction candidate is formed when the decision to replace with the correction candidate is executed. This allows the user to print the secondary document at any timing. When the user operates the "do not modify" button, the above printing and storage are not executed. Also, when the user operates the button for "manual correction", the user can change the desired visibility for each graph.

It should be noted that information regarding the estimation result in step S910 may be fed back to the learning model as teacher data when the button for "correct to correction candidate" is operated. In this case, it means that the estimation result in step S910 was correct. This feedback allows the learning model to improve its estimated accuracy rate. The timing of the feedback is not particularly limited, and can be, for example, when the button for "correct to correction candidate" is operated. Information relating to the estimation result to be fed back is not particularly limited, and may be input data X or the like, for example. The processing on the UI display unit 401 for such graphs can be performed similarly for tables.

Further, in the present embodiment, the primary document is a color-printed document and the secondary document is a monochrome-printed document, but the present invention is not limited to this. For example, the primary document may be a color printed document, and the secondary document may also be a color printed document. In this case, for example, for a person with color blindness who has difficulty in distinguishing between red and green colors, the difficulty in distinguishing colors can be resolved by changing the colors of the graphs and tables in the secondary manuscript. be able to. Also, the primary document may be a monochrome printed document, and the secondary document may also be a monochrome printed document. In this case, for graphs and tables that were difficult to identify in the primary manuscript due to monochrome printing, by changing the hatching of each graph and table in the secondary manuscript, each graph and table can be distinguished. can be improved.

The disclosure of this embodiment includes the following configurations, methods and programs.
(Configuration 1) An image forming apparatus for forming an image,
reading means for reading an original;
A manuscript read by the reading means using a machine-learned trained model for estimating at least one of a graph area containing a graph and a table area containing a table in the manuscript. and estimation processing means for estimating the graphic area in the image forming apparatus.
(Arrangement 2) When the graph area is estimated as the diagram area, the estimation processing means can extract the graph within the graph area and change the visibility of the graph. The image forming apparatus according to Arrangement 1, wherein when the table area is estimated as , the table in the table area is extracted and the visibility of the table can be changed.
(Configuration 3) the graph area includes a plurality of the graphs, the table area includes a plurality of the tables,
The image forming apparatus according to Structure 2, wherein the estimation processing means makes the visibility different for each graph and makes the visibility different for each table.
(Configuration 4) The image according to configuration 2 or 3, wherein the change in visibility of the graph and the table includes at least one of color change, shading change, and hatching change. forming device.
(Arrangement 5) The image forming apparatus according to any one of Arrangements 2 to 4, further comprising display means capable of displaying a sample image of the graphic area with the visibility changed.
(Structure 6) The image forming apparatus according to structure 5, wherein the display means can display the graphic area in a state before the visibility is changed, side by side with the sample image.
(Arrangement 7) Provided with operating means for receiving an operation from a user who uses the image forming apparatus,
The image forming apparatus according to configuration 6, wherein the operation means is capable of determining to replace the graphic area of the document read by the reading means with the sample image. .
(Structure 8) The image forming apparatus according to structure 7, further comprising printing means capable of printing a document on which an image including the sample image is formed when the determination is made by the operation means.
(Arrangement 9) The document read by the reading means is a document in which at least one of the graph and the table is printed in color,
The image forming apparatus according to Arrangement 8, wherein the document on which the image including the sample image is formed is a document printed in monochrome.
(Structure 10) Any one of structures 7 to 9, further comprising storage means capable of storing data of a document on which an image including the sample image is formed when the determination is made by the operation means. 1. The image forming apparatus according to item 1.
(Arrangement 11) The trained model is a neural network, and has as input data information relating to the graph and the table of the document from which the figure/table area is to be estimated, and output data as to whether or not the figure/table area exists. The image forming apparatus according to any one of Configurations 1 to 10.
(Arrangement 12) Provided with operating means for receiving an operation from a user who uses the image forming apparatus,
12. The image forming apparatus according to any one of configurations 1 to 11, wherein the reading unit operates in response to an operation from the user via the operation unit.
(Structure 13) A method for controlling an image forming apparatus that forms an image, comprising:
a reading process for reading an original;
A manuscript read by the reading means using a machine-learned trained model for estimating at least one of a graph area containing a graph and a table area containing a table in the manuscript. and an estimation processing step of estimating the graphic area in the image forming apparatus.
(Arrangement 14) A program for causing a computer to execute each means of the image forming apparatus according to any one of Arrangements 1 to 12.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications and changes are possible within the scope of the gist thereof. The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors of the computer of the system or device reads the program. It can also be realized by executing processing. The invention can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

10 scanner unit (reading means)
20 printer section (printing means)
101 image forming apparatus (image processing apparatus)
102 machine learning server 140 operation unit (operation means)
401 UI display unit 403 JOB control unit 404 Image reading unit 405 Estimation processing unit 414 Machine learning unit

Claims (14)

An image forming apparatus for forming an image,
reading means for reading an original;
A manuscript read by the reading means using a machine-learned trained model for estimating at least one of a graph area containing a graph and a table area containing a table in the manuscript. and estimation processing means for estimating the graphic area in the image forming apparatus. When the graph area is estimated as the diagram area, the estimation processing means can extract the graph within the graph area and change the visibility of the graph. 2. The image forming apparatus according to claim 1, wherein the table in the table area can be extracted and the visibility of the table can be changed when the is estimated. the graph area includes a plurality of the graphs, the table area includes a plurality of the tables,
3. The image forming apparatus according to claim 2, wherein the estimation processing means makes the visibility different for each graph and makes the visibility different for each table. 3. The image forming apparatus according to claim 2, wherein the change in visibility of the graph and the table includes at least one of color change, shade change, and hatching change. 3. The image forming apparatus according to claim 2, further comprising display means capable of displaying a sample image of said graphic area with said visibility changed. 6. The image forming apparatus according to claim 5, wherein the display unit can display the graphic area in a state before the visibility is changed, side by side with the sample image. an operating means for receiving an operation from a user who uses the image forming apparatus;
7. The image forming apparatus according to claim 6, wherein said operation means can execute a decision to replace said graphic area of said original document read by said reading means with said sample image. Device. 8. The image forming apparatus according to claim 7, further comprising printing means capable of printing a document on which an image including said sample image is formed when said decision is made by said operation means. The document read by the reading means is a document in which at least one of the graph and the table is printed in color,
9. The image forming apparatus according to claim 8, wherein the document on which the image including the sample image is formed is a document printed in monochrome. 8. The image forming apparatus according to claim 7, further comprising storage means for storing data of a document on which an image including said sample image is formed when said decision is made by said operation means. 3. The trained model is a neural network, and has as input data information relating to the graph and the table of the document from which the figure/table area is to be estimated, and output data as to whether or not the figure/table area exists. 1. The image forming apparatus according to 1. an operating means for receiving an operation from a user who uses the image forming apparatus;
2. The image forming apparatus according to claim 1, wherein said reading means operates according to an operation by said user via said operation means. A method of controlling an image forming apparatus for forming an image, comprising:
a reading process for reading an original;
A document read in the reading step using a trained model that has undergone machine learning for estimating at least one of a graph area including a graph and a table area including a table in the document. and an estimation processing step of estimating the graphic area in the image forming apparatus. A program for causing a computer to execute each means of the image forming apparatus according to claim 1 .

Images