JP6968241B1 - Information processing equipment, information processing methods and programs - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus, an information processing method and a program for analyzing a wider variety of graph images. SOLUTION: A graph image analysis device 1 has a graph image acquisition unit for acquiring a graph image, a graph classification unit for classifying the acquired graph image for each graph type, and a graph image classified for each graph type. A probability map generator that generates different types of probability maps, a component extractor that extracts the components in the graph image based on the generated probability map, and a value extractor that extracts the values of the components in the extracted graph image. It includes a unit and an output unit that outputs the value of the component extracted by the value extraction unit. [Selection diagram] Fig. 1

Description

The present invention relates to an information processing apparatus, an information processing method and a program, and more particularly to a technique for analyzing an image of a graph.

A technique for converting an image including a line graph into digital data is known.
For example, Patent Document 1 analyzes a line graph display of collected patient data to identify individual pixels and to provide a plurality of pixel display data elements that determine the relative position of each pixel with respect to a reference point. Disclose a processing system for image display data generated in a format and stored in a file.

Patent Document 2 reads a non-digital vibration waveform recorded on paper or the like by a scanner, reads amplitude data for each set pitch of the read graph image, and converts the read graph into digital data that can be used by a shaking table. Disclose the data digitizer.

Non-Patent Document 1 discloses a technique for analyzing a line graph by extracting a chart from a PDF file of a research paper, separating the line graph from the extracted chart, and detecting the axis and the legend of the line graph.

Japanese Patent Publication No. 2005-503851 Japanese Patent Application Laid-Open No. 2002-328068 Siegel, N.M. , Horvitz, Z. et al. , Levin, R.M. , Divvala, S.A. , Farhadi, A. "FigureSeer: Parsing Resist-Figures in Research Papers" ECCV (2016) pp. 664-680

However, there are various types of graphs such as line graphs, bar graphs, and pie charts. However, the techniques in the above-mentioned prior art literature limit the analysis target to line graphs only, and other graphs. It is not versatile because it cannot analyze type graphs.

Further, in the techniques of Patent Document 1 to Patent Document 2, when tracing a line graph, the user must specify the color and luminance amplitude of the graph, or set the X-axis, Y-axis, and scale. However, when trying to analyze a large number of graph images, it imposes an excessive load on the user.
On the other hand, in the technique of Non-Patent Document 1, the line graph is traced by detecting the legend located in the margin of the line graph and detecting the line type of the symbol described in the detected legend. It is not supposed to analyze line graph images that do not include legends.

The present invention has been made to solve the above problems, and an object of the present invention is to provide an information processing apparatus, an information processing method, and a program capable of analyzing a wider variety of graph images.

In order to solve the above problems, one aspect of the information processing apparatus according to the present invention classifies a graph image acquisition unit for acquiring a graph image and the graph image acquired by the graph image acquisition unit for each graph type. The graph classification unit, the probability map generation unit that generates different types of probability maps for each graph type classified by the graph classification unit from the graph image, and the probability map generated by the probability map generation unit. Based on this, a component extraction unit that extracts components in the graph image, a value extraction unit that extracts the value of the component in the graph image extracted by the component extraction unit, and the value extraction unit that extracts the value of the component. It includes an output unit that outputs the value of the component.

The probability map generation unit may generate the probability map of the graph image by using a neural network.

The neural network may have different output channels for each graph type.

The probability map generation unit may generate a plurality of different types of probability maps for separating different components for each of the graph types.

The component extractor calculates the cost of each pixel of the graph image of the line graph using the first kind of probability map for the line graph, and traces the line graph on the pixel with the minimum cost. You can do it.

The component extraction unit may calculate the cost of each pixel of the graph image of the line graph by using dynamic programming.

The component extractor may calculate the cost based on a function that evaluates the smoothness of the lines in the line graph.

The component extractor may calculate the cost based on a function that evaluates the consistency of the line color or line type of the line graph.

The component extraction unit detects the start point and the end point of the line graph by stacking the existence probabilities of the line graph in the Y coordinate axis direction at each X coordinate position using the first type of probability map. The line graph may be traced from the detected start point to the end point.

The component extractor may trace the graph by concatenating adjacent pixels using a second type of probability map for graphs other than line graphs.

The component extraction unit may connect the adjacent pixels of the graph image of the graph by using CCA (Connected Component Analysis).

The component extraction unit may correct the extracted component with reference to the second type of probability map.

One aspect of the information processing method according to the present invention is an information processing method executed by an information processing apparatus, which includes a step of acquiring a graph image, a step of classifying the acquired graph image for each graph type, and the above. From the graph image, a step of generating a different kind of probability map for each of the classified graph types, a step of extracting components in the graph image based on the generated probability map, and the extracted graph. It includes a step of extracting the value of the component of the image and a step of outputting the value of the extracted component.

One aspect of the information processing program according to the present invention is an information processing program for causing a computer to execute information processing, and the program includes a graph image acquisition process for acquiring a graph image and the graph image. A graph classification process that classifies the graph image acquired by the acquisition process for each graph type, and a probability map generation that generates a different type of probability map for each graph type classified by the graph classification process from the graph image. The processing, the component extraction process for extracting the components in the graph image based on the probability map generated by the probability map generation unit, and the value of the component in the graph image extracted by the component extraction unit. The purpose is to execute a process including a value extraction process for extracting and an output process for outputting the value of the component extracted by the value extraction unit.

According to the present invention, it is possible to analyze images of a wider variety of graphs.
The above-mentioned object, aspect and effect of the present invention and the above-mentioned object, aspect and effect of the present invention not described above are to be used by those skilled in the art to carry out the following invention by referring to the accompanying drawings and the description of the scope of claims. It can be understood from the form of.

FIG. 1 is a block diagram showing an example of the functional configuration of the graph image analysis apparatus according to each embodiment of the present invention. FIG. 2 is a conceptual diagram illustrating an outline of a graph image analysis process executed by the graph image analysis apparatus according to each embodiment. FIG. 3 is a flowchart showing an example of a processing procedure of the graph image analysis process executed by the graph image analysis apparatus according to each embodiment. FIG. 4 is a diagram showing an example of an output channel of a neural network output by a probability map generation unit of a graph image analysis device for a bar graph. FIG. 5 is a diagram showing an example of an output channel of a neural network output by a probability map generation unit of a graph image analysis device for a pie chart. FIG. 6 is a diagram showing an example of an output channel of a neural network output by a probability map generation unit of a graph image analysis device for a line graph. FIG. 7 is a flowchart showing an example of a detailed processing procedure of the component extraction process of S4 of FIG. 3 executed by the component extraction unit of the graph image analysis device for the bar graph and the pie chart. FIG. 8 is a diagram illustrating an example in which the component extraction unit of the graph image analysis device extracts the components of the bar graph from the bar graph image using the probability map of the bar graph. FIG. 9 is a flowchart showing an example of a detailed processing procedure of the component extraction process of S4 of FIG. 3 executed by the component extraction unit of the graph image analysis device for the line graph. FIG. 10 is a diagram illustrating an example of a line graph traced from a line graph image by a component extraction unit of the graph image analysis device using a probability map of the line graph. FIG. 11 is a diagram illustrating an example in which the value extraction unit of the graph image analysis device extracts graph values from a line graph image. FIG. 12 is a block diagram showing an example of the hardware configuration of the graph image analysis apparatus according to each embodiment of the present invention.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. Among the components disclosed below, those having the same function are designated by the same reference numerals, and the description thereof will be omitted. The embodiments disclosed below are examples as means for realizing the present invention, and should be appropriately modified or modified depending on the configuration of the apparatus to which the present invention is applied and various conditions, and the present invention is described below. Is not limited to the embodiment of the above. Moreover, not all combinations of features described in the present embodiment are essential for the means of solving the present invention.

The graph image analysis device according to the present embodiment classifies the input graph image for each graph type, generates a different probability map for each classified graph type, and automatically generates the graph image using the generated probability map. Analyze.
In the following, the graph image analyzer classifies the input graph image into graph types of bar graph, circle graph, and line graph, generates a probability map for each graph type, and uses the generated probability map. , An example of analyzing each type of graph image will be described.

However, this embodiment is not limited to this. The graph image analyzer may classify and analyze any other graph image, such as a flow chart, a raider chart, a scatter plot, a Venn diagram, or the like.
In the present embodiment, the graph image means an image including a graph (a chart or a chart, hereinafter collectively referred to as a "graph") in all or a part of the image. The graph image may be input to the graph image analyzer in any format.

<Functional configuration of graph image analysis device>
FIG. 1 is a block diagram showing an example of the functional configuration of the graph image analysis device 1 according to the present embodiment.
The graph image analysis device 1 shown in FIG. 1 includes a data acquisition unit 11, a graph classification unit 12, a probability map generation unit 13, a component extraction unit 14, a value extraction unit 15, and an output unit 16.

The data acquisition unit 11 acquires input data from the input data set 3 and supplies the acquired input data to the graph classification unit 12. The input data acquired by the data acquisition unit 11 may include a graph image. The input data set 3 consists of a non-volatile storage device, which may be locally provided or remotely connected to the graph image analysis device 1.
The graph image may have any image file format such as a bitmap, GIF (Graphics Interchange Form), JPEG (Joint Photographic Experts Group), TIFF (Tag Image File Form) and the like. When a graph is included in a part of the image, the image may have an arbitrary file format such as PDF (Portable Document Form) or HTML (HyperText Markup Language) constituting a web page.

The data acquisition unit 11 may receive the input data from the opposite device via the communication I / F instead of the input data set 3.
The data acquisition unit 11 may also accept input of various parameters necessary for executing the graph image analysis process in the graph image analysis device 1.

The graph classification unit 12 extracts a graph image to be analyzed in the subsequent stage from the input data supplied from the data acquisition unit 11, classifies the extracted graph image into one of a plurality of graph types, and classifies the extracted graph image. The graph image is supplied to the probability map generation unit 13. The graph classification unit 12 may extract, for example, a bar graph, a pie chart, a line graph, or the like as a graph image to be analyzed from the input data.
In the present embodiment, the graph classification unit 12 may extract a graph image from the input data and classify the extracted graph image by using a classification network such as a neural network.

The probability map generation unit 13 generates a probability map of the graph image for each classified graph type with respect to the classified graph image supplied from the graph classification unit 12, and the generated probability map is a component extraction unit. Supply to 14.
In the present embodiment, the probability map (Probability Map) generated by the probability map generation unit 13 estimates and estimates the probability that a component constituting a classified type graph appears for each pixel of the graph image. It is a map composed of probability distributions.
The probability map generation unit 13 may generate a probability map from the classified graph images using, for example, a network for generating a probability map such as a neural network. The probability map generator 13 may also generate different types of probability maps for each classified graph type (see FIG. 2 below), and for each classified graph type, different types of probability maps. May be generated (see FIGS. 4-6 below).

The component extraction unit 14 extracts a component (hereinafter, simply referred to as “component”) constituting the graph from the graph image by using the probability map supplied from the probability map generation unit 13.
Specifically, the component extraction unit 14 uses the probability map supplied from the probability map generation unit 13 corresponding to the type in which the graph image is classified to the graph image classified by the graph classification unit 13. Separate each of the multiple components that make up the graph from the graph image.

The component extraction unit 14 may separate the main component from the plurality of components constituting the graph from the graph image. Multiple types of probability maps may be generated for each of the multiple key components to be separated.
The component extraction unit 14 traces a graph for each of the plurality of components separated from the graph image, and supplies the separated components and the graph to the value extraction unit 15.

The value extraction unit 15 extracts the corresponding graph value for each component of the graph image supplied from the component extraction unit 14.
Specifically, the value extraction unit 15 extracts the labels of the axes of the graph image and the labels attached to each component, and based on the text and geometric arrangement of the extracted labels, gives each component of the graph image. Generate the corresponding graph values.

The graph itself included in the graph image is a visualization of predetermined input data, but usually, the original input data on which the graph is drawn cannot be directly accessed from the imaged graph image.
In the present embodiment, the value extraction unit 15 recovers the original input data on which the graph is drawn by generating the graph values corresponding to each component of the graph image.
The value extraction unit 15 may continuously generate graph values corresponding to consecutive pixels of each component of the graph image, or may generate graph values discretely at predetermined pixel intervals. good.
The value extraction unit 15 adds or associates the generated graph value to each component of the graph image, and supplies the value to the output unit 16.

The output unit 16 synthesizes each component of the graph image extracted by the component extraction unit 14 into a graph and outputs it as digital data to the output data set 4. The graph output to the output data set 4 may be output to the outside in a push type or a pull type via the user interface of the graph image analysis device 1 or another device.
The values of the graph extracted by the value extraction unit 15 are added to each component of the graph synthesized by the output unit 16. Therefore, the graph image analysis device 1 or another device can read the graph data synthesized from the output data set 4 as post-processing of the graph image analysis process, process it arbitrarily, and output it.
For example, the output unit 16 or the post-processing may change the graph type to a graph type different from the graph type classified by the graph classification unit 12, and output the graph, and also scale, the range of graphing, and the color of the graph component. You may output the graph by changing the line type and the like.
The output unit 16 may independently output the value of the graph generated by the value extraction unit 15 without adding it to each component of the graph image. Alternatively, the output unit 16 may format the values of these graphs into an arbitrary format such as a table format and output the graph.

The graph image analysis device 1 is communicably connected to a client device composed of a PC (Personal Computer) or the like, and the client device is used when the graph image analysis device 1 executes input / output of information with the outside. A user interface may be provided, and some or all of the components 11 to 16 of the graph image analyzer 1 may be provided.

FIG. 2 is a conceptual diagram illustrating an outline of the graph image analysis process executed by the graph image analysis device 1 according to the present embodiment.
In the present embodiment, as a first step, the graph image analysis device 1 extracts a graph image to be analyzed from the input data and classifies the extracted graph image into one of a plurality of graph types.
With reference to FIG. 2, the input data 21 is input to the classification network 22 composed of a neural network or the like, and the classification network 22 includes a classification result including graph images classified into any graph type. 23 is output. The classification result 23 output by the classification network 22 includes graph images of a line graph (Line), a pie graph (Pie), a horizontal bar graph (H-Bar), and a vertical bar graph (V-Bar). The classification network 22 is trained in advance using various graph image data and graph type labels as training data.
Each of these classified graph images is input to the network 24 for generating a probability map in the subsequent stage. On the other hand, the network 22 for classification also outputs graph images (for example, map graphs, etc.) of types other than the graph images to be analyzed, photographs, etc. as classification results, but these classification results are the probability of the latter stage. It is not input to the network 24 for map generation.

In the present embodiment, as the second step, the graph image analysis device 1 generates a different type of probability map for each classified graph type, and uses the generated probability map to generate a component of the graph image from the graph image. Separate and trace the graph with the components of the separated graph image, and extract the values of the graph corresponding to the components of the traced graph.

With reference to FIG. 2, the classification result 23, that is, the data of the classified graph image is input to the probability map generation network 24 composed of a neural network or the like, and the probability map generation network 24 is classified. The probability maps 25a to 25d are output for each graph type.
In FIG. 2, the probability map output by the network 24 for generating the probability map includes the probability map 25a of the line graph, the probability map 25b of the pie chart, the probability map 25c of the horizontal bar graph, and the probability map 25d of the vertical bar graph. The details of the configuration of the probability map will be described later with reference to FIGS. 4 to 6.
The network 24 for generating a probability map uses graph image data to which a classified graph type is added and labels corresponding to graph types such as color, borderline, segmentation, legend, and line type as training data in advance. Let me learn.
The classification network 22 and the probability map generation network 24 may be a neural network such as CNN (Convolutional Neural Network), or may use another machine learning model such as SVM (Support Vector Machine). good.

In the probability maps 25a to 25d of FIG. 2, the pixels whose likelihood of appearance of the main component of the graph image is equal to or higher than a predetermined threshold value are displayed in white. The line graph probability map 25a separates the components of each line graph. The pie chart probability map 25b separates the components of each pie slice in the pie chart. The probability map 25c of the horizontal bar graph and the probability map 25d of the vertical bar graph separate the components of each bar segment, respectively.

In the present embodiment, each component is separated and extracted from the graph image by using different types of probability maps 25a to 25d for each of these graph types. With reference to FIG. 2, for the graph image of the line graph, the component 26a of each line graph is extracted and traced using the probability map 25a of the line graph. For the graph image of the pie chart, the component 26b of each pie slice is extracted and traced using the probability map 25b of the pie chart. For the graph images of the horizontal bar graph and the vertical bar graph, the components 26c and 26d of each bar segment are extracted and traced using the probability map 25c of the horizontal bar graph and the probability map 25d of the vertical bar graph, respectively.

Graph values are extracted for each of the components 26a-26d extracted from the graph image. Specifically, the area of the coordinate axis label of each graph image of the line graph, pie chart, horizontal bar graph, and column graph, the label of each component, the title, the legend, etc. is shown by the rectangular area in the graph images 27a to 27d. The text is detected and the characters are recognized. Graph values are generated based on the result of this character recognition and the geometric arrangement of each component extracted on the graph image. The generated graph values are added to each component of the traced graph, and the digitized graph data 28 is output.

<Processing procedure for graph image analysis processing>
FIG. 3 is a flowchart showing an example of a processing procedure of the graph image analysis process executed by the graph image analysis device 1 according to the present embodiment.
Each step in FIG. 3 is realized by the CPU reading and executing a program stored in a storage device such as an HDD of the graph image analysis device 1. Further, at least a part of the flowchart shown in FIG. 3 may be realized by hardware. When it is realized by hardware, for example, by using a predetermined compiler, a dedicated circuit may be automatically generated on FPGA (Field Programmable Gate Array) from a program for realizing each step. Further, a Gate Array circuit may be formed in the same manner as the FPGA and realized as hardware. Further, it may be realized by ASIC (Application Specific Integrated Circuit).

In S1, the data acquisition unit 11 of the graph image analysis device 1 acquires the input data including the graph image from the input data set 3.
In S2, the graph classification unit 12 of the graph image analysis device 1 extracts the graph image to be analyzed from the input data using the classification network 22, and the extracted graph image is used as a plurality of graphs to be analyzed. Classify into one of the types.

In S3, the probability map generation unit 13 of the graph image analysis device 1 uses the network 24 for generating the probability map to generate a probability map for extracting each component from the classified graph images for each graph type. ..
The network 24 for generating a probability map is formed as a learning model that has been trained with different training data for each graph type to be analyzed, but only the output channel of the network needs to be different for each graph type, other than the output channel. The network may have a common structure. This improves the versatility of the network 24 for generating the probability map.

FIG. 4 is a diagram showing an example of an output channel of the network 24 for generating a probability map, which generates a probability map for extracting each component from the graph image of the column graph.
When the graph image of the vertical bar graph shown in FIG. 4A is used as input data, the network 24 for generating a probability map has an output channel of a color (main color) for coloring the bar segment shown in FIG. 4B. The segmentation output channel that separates each bar segment shown in FIG. 4 (c), the borderline output channel that defines the boundary of each bar segment shown in FIG. 4 (d), and the legend shown in FIG. 4 (e). Each has an output channel of. The probability map generation unit 13 generates a plurality of types of probability maps corresponding to FIGS. 4 (b) to 4 (e).

FIG. 5 is a diagram showing an example of an output channel of the network 24 for generating a probability map, which generates a probability map for extracting each component from a graph image of a pie chart.
When the graph image of the pie chart shown in FIG. 5 (a) is used as the input data, the network 24 for generating the probability map has the output channel of the color (main color) for coloring the pie slice shown in FIG. 5 (b). The segmentation output channel that separates each pie slice shown in FIG. 5 (c), the borderline output channel that defines the boundary of each pie slice shown in dI, and the legend shown in FIG. 5 (e). Each includes an output channel. The probability map generation unit 13 generates a plurality of types of probability maps corresponding to FIGS. 5 (b) to 5 (e).

On the other hand, FIG. 6 is a diagram showing an example of an output channel of the network 24 for generating a probability map, which generates a probability map for extracting each component from the graph image of the line graph.
When the graph image of the line graph shown in FIG. 6 (a) is used as input data, the network 24 for generating the probability map has the output channel of the solid line graph shown in FIG. 6 (b) and the figure for each line type. The output channel of the dotted line graph shown in 6 (c), the output channel of the broken line line graph shown in FIG. 6 (d), the output channel of the alternate long and short dash line graph shown in FIG. 6 (e), and the figure. Each of the output channels of the legend shown in 6 (f) is provided. The probability map generation unit 13 generates a plurality of types of probability maps corresponding to FIGS. 6 (b) to 6 (f).

The configuration of the output channel (probability map) of the network 24 for generating the probability map shown in FIGS. 4 to 6 is an example, and the present embodiment is not limited to this. For example, the output channels (probability maps) shown in FIGS. 4 (e), 5 (e), and 6 (f) for extracting a legend from a graph image mask the area of the legend and are used from the graph image. It may be used to define the area where the graph is drawn, but it does not necessarily have to be generated. The probability map generation unit 13 may generate a plurality of different types of probability maps according to the graph type so as to facilitate tracing of the components of the graph.

Returning to FIG. 3, the component extraction unit 14 of the graph image analysis device 1 estimated each component of the graph from the graph image using the probability map generated corresponding to each graph type in S3, and estimated. Trace the graph component.
The component extraction unit 14 may estimate each component of the graph from the graph image by a method different for each graph type. Details of the component extraction process of these graphs will be described later with reference to FIGS. 7 to 11.

In this embodiment, when estimating the components of the graph from the graph image, the probability map provided for each graph type is used. By using the probability map provided for each graph type, the probability that each component appears can be obtained for each pixel on the graph image, and the estimation of the area of each component is facilitated.
So, for example, if the graph image contains artifacts, noise or gradients, if the components of the graph intersect or overlap or are separated from other components, if they are highly compressed and the resolution is low, then the boundaries of the graph components will be. Even if it is not clear, each component of the graph can be estimated with high accuracy from the graph image.

In S5, the value extraction unit 15 of the graph image analysis device 1 extracts the value of the graph corresponding to each component of the graph estimated in S4.
The value extraction unit 15 describes the pie chart graph image based on the area of the pie slice segment (component) acquired in S4 and the character recognition result of the text labeled on the axis of the graph and the component. Extract the value of.
The value extraction unit 15 also geometrically relates the graph image of the bar graph and the line graph to the pixel coordinates of the components of the bar graph and the line graph and the character recognition result of the text labeled on the axis and the component of the graph. The value of each component of the graph is extracted by making a judgment based on the geometry.

More specifically, the value extraction unit 15 first reads the entire graph image by OCR (Optical Character Recognition), detects the text, and displays the text box of the detected text on either the vertical axis or the horizontal axis. Group to. Text boxes may also be grouped into legends or titles.
The value extraction unit 15 then determines the scale of the vertical and horizontal coordinate axes for the bar graph and the line graph from the position of the text box containing the numbers as text.
Applying the scale of the determined coordinate axes, the value extraction unit 15 converts the coordinates of each component of the graph into the values of the graph.

Specifically, the value extraction unit 15 converts the pixel coordinates of all the points on the component of each line of the line graph into the values of the graph. The value extraction unit 15 also converts the vertex Y coordinate (vertical coordinate) and the right end X coordinate (abscissa) of the component of each bar segment of the bar graph into the graph value.
On the other hand, for a pie chart, the value extraction unit 15 generates graph values based on the area ratio of the components of each pie slice in the pie chart.

In S6, the output unit 16 of the graph image analysis device 1 adds information such as graph values and coordinate axis scale information extracted by the value extraction unit 15 to the traced graph components extracted by the component extraction unit 14. Then, it is output to the output data set 4 as the digital data of the graph.

(First graph component extraction process)
FIG. 7 is a flowchart showing an example of a detailed processing procedure of the component extraction (graph trace) process of S4 of FIG. 3 executed by the component extraction unit 14 of the graph image analysis device 1 for a bar graph and a pie chart.
In S401, the component extraction unit 14 masks the legend area in the graph image. The component extraction unit 14 detects the legend area from the graph image using the probability map of the legend generated in S3 of FIG. 3, and the detected legend area is the segmentation generated in S3 of FIG. You may mask it on the probability map. This makes it possible to distinguish between the area where the graph component exists and the area of the legend in the graph image.

In S402, the component extraction unit 14 defines a borderline for each component in the graph image. Specifically, the component extraction unit 14 defines the borderline using the borderline probability map generated in S3 of FIG. 3, and removes the borer line from the segmentation probability map of the graph image.
In S403, the component extraction unit 14 applies CCA (Connected Component Analysis) to extract components of a bar graph and a pie chart, and connects adjacent pixels in the graph image.
CCA can connect pixels and extract components by grouping adjacent pixels with similar colors in the RGB color space.

In S404, the component extraction unit 14 uses the segmentation probability map and the color probability map from which the border line has been removed in S402, and applies CCA in S403 to concatenate the concatenated components as the components of the graph of the graph image. Extract and trace the components of the graph.
Further, the component extraction unit 14 may discard a component that deviates from the shape of the component estimated from the segmentation probability map for each graph type among the connected components configured by connecting the pixels.
For example, in the case of a bar graph, the non-rectangular concatenated component may be discarded, and in the case of a pie chart, the non-pie slice concatenated component may be discarded.

In S405, the component extraction unit 14 corrects the components extracted in S404 as necessary.
In S403 and S404, adjacent pixels are connected when they have similar colors to each other. Therefore, for example, when the component of the graph has a color gradation or texture instead of a single color, or a 3D effect is given. In some cases, the pixels that should be connected may not be connected.
In this embodiment, since the probability map of the color (main color) generated in S3 of FIG. 3 and the probability map of segmentation are used, the color and shape of the components of the graph estimated from the probability map are extracted in S404. It is possible to correct the components of the graph.

FIG. 8 is a diagram illustrating an example in which the component extraction unit 14 of the graph image analysis device 1 extracts a bar graph component from a bar graph image using a bar graph probability map.
With reference to FIG. 8, the horizontal bar graph displays the year on the Y coordinate axis and the oil production on the X coordinate axis, and each year's oil production constitutes one bar segment and one bar segment. Is further broken down into each subsegment of regions A to F and other regions.
By executing the component extraction process for the bar graph, the component extraction unit 14 extracts each of the six bar segments as a component, as can be understood from the fact that each subsegment is surrounded by a thick frame. ..

For example, it is assumed that a part of the bar segment of a certain year in FIG. 8 is not divided into subsegments of each region at the stage where the processes of S401 to S404 of FIG. 7 are executed. In the present embodiment, by executing the correction process of S405 of FIG. 7, a plurality of components that are difficult to separate due to their similarities in color are separated from each other by using the color probability map and the segmentation probability map. Can be extracted to.

(Second graph component extraction process)
FIG. 9 is a flowchart showing an example of a detailed processing procedure of the component extraction (graph trace) process of S4 of FIG. 3 executed by the component extraction unit 14 of the graph image analysis device 1 for the line graph.
In S411, the component extraction unit 14 of the graph image analysis device 1 calculates the existence range of the line graph.

Specifically, the component extraction unit 14 detects the start point and the end point of the line graph by acquiring the X coordinate range in which the line graph exists. In S411, the component extraction unit 14 averages the probability maps of the plurality of line types generated in S3 of FIG. 3 (four types of line types in FIGS. 6 (b) to 6 (e)) for each pixel position. It is assumed that the line probability map has been calculated in advance.
The component extraction unit 14 calculates the existence probability of the stacked lines in each X coordinate axis by calculating the sum of the existence probabilities along the Y coordinate axis direction. Next, in order to acquire the existence distribution of the lines on the X coordinate axis, the component extraction unit 14 binarizes the existence probabilities of the stacked lines using _{the threshold value θ th.} In this way, the existence range of the line graph is acquired as the minimum and maximum values on the X coordinate axis of the cluster of the existence distribution of the line. As described above, in the present embodiment, it is not necessary to define the existence range of the line graph by referring to the area of the legend, and even if the graph image of the line graph does not include the legend outside the existence range of the line graph, the line You can trace the graph.

In S412, the component extraction unit 14 counts the number of line graphs.
Specifically, the component extracting unit 14 binarizes the line probability map using the threshold value p _th, lines probability map that is binarized in the X-coordinate position, by scanning along the Y coordinate axis Counts the number of pixel clusters on the X coordinate axis. Thus, the number of line graphs is acquired as the number of pixel clusters on the X coordinate axis.

In S413, the component extraction unit 14 applies a dynamic programming method (DP) to calculate the cost of adjacent pixels.
In S414, the component extraction unit 14 traces a line on the pixel with the lowest cost in S413.

CCA, which connects adjacent pixels based on the color in the RGB color space, is difficult to apply to a line graph that draws a line with dots. In this embodiment, a line graph is traced using a dynamic programming method.
Dynamic programming (DP) is an algorithm that divides a target problem into a plurality of subproblems and solves them while recording the calculation results of the subproblems.
In this embodiment, the dynamic programming method is applied to calculate the cost of each pixel within the existence range of the line graph. By reusing the calculation result in the middle, it is possible to avoid the repetition of the calculation and improve the calculation efficiency.

The component extraction unit 14 uses the existence range of the line graph calculated in S411 and the number of count line graphs in S412 to perform dynamic programming along the X-axis from the start point to the end point of the detected line graph. Trace the line graph with.
Specifically, the component extraction unit 14 uses the table W to hold the total cost of the points (pixels) on the traced line. The table W holds each value of the coordinates x = (x, y).
The value of W is calculated by the following equation 1.

（式１）

(Equation 1)

Here, x represents the pixel coordinates as X = (i, j). _xprev is the immediately preceding pixel (the final point traced immediately before) on the x coordinate of x.
X- _prev is a pixel group of x-coordinates up to the immediately preceding pixel i-1, and is represented by the following equation 2.

（式２）

(Equation 2)

In order to determine which traced line x belongs to, the component extractor 14 _{finds the best from the X rev of} Equation 2 that minimizes the value of the cost function in Equation 1 by dynamic programming. ..

In the present embodiment, a plurality of evaluation functions, for example, as shown in equation 1, as five of the evaluation _{function, E color, E pattern, E} smooth, E prob, and using _{E DUPL,} these evaluation functions Calculate the cost of each point (pixel) so that the cost evaluated in is minimized.
The evaluation function _Ecolor is a color function that maintains consistency in the color of the traced line. Since the information of the traced line to which xprev belongs is held by the dynamic programming method, the average line color of xprev can be calculated from the information of this line. E _color is calculated by the color difference between the average line color and the pixel color of x.

The evaluation function E _pattern is a function of a line type that maintains consistency in the line type (line pattern) of the traced line. The evaluation function E _pattern is calculated by the difference between the average line type of xprev and the line type of x, as in the evaluation function E _color.
The evaluation function E _smooth is a function of line smoothness for smoothly connecting pixels adjacent to each other of traced lines without an excessively steep slope. Evaluation function _{E smooth smooth} is calculated by the absolute difference of the slope of xprev slope and x.

The evaluation function E _prob is a function of existence probability for connecting traced lines along a pixel having a high existence probability value. The evaluation function E _prob is calculated by the average value (mean value) of the existence probabilities of a plurality of line types.
The evaluation function E _double is a function of duplication that gives a penalty when a plurality of lines are traced on the same pixel. If x has already been traced on another line, the _merit function E double returns 1, otherwise it returns 0.
λ1 to λ5 are weighting coefficients of each evaluation function.

The component extraction unit 14 uses a plurality of evaluation functions as described above, and at each point of the X coordinate from the start point to the end point of the detected line graph, the minimum cost as the next point (pixel) to be traced. Points (pixels) are determined in order.
In the present embodiment, for example, the probability map of the line graph generated in S3 of FIG. 3 by _{the evaluation function E pattern of the line type, the evaluation function E prob} of the existence probability, etc. (FIGS. 6 (b) to 6 (e). ) May refer to the probability map of four types of line types). For example, it can be determined that a point (pixel) that does not fit into any of the four line types is not traced.
As described above, in the present embodiment, the evaluation function considering the probability map of the line graph and the attributes of the line graph such as the line type and color consistency of the line graph and the smoothness of the line is used by the dynamic programming method. Therefore, the line graph can be traced with high accuracy and less computational load without relying on the legend information in the graph image.
Alternatively, the component extraction unit 14 does not use the dynamic programming method using the above evaluation function or the probability map, and the line type and color of the line graph are consistent between the start point and the end point of the detected line graph. The line graph included in the graph image may be traced based on the attributes of the line graph such as sex and line smoothness.

In S415, the component extraction unit 14 determines whether or not the traced line graph has reached the end point detected in S411. While the traced line graph does not reach the end point (S415: N), it returns to S413, and the component extraction unit 14 repeats the processes of S413 and S414. On the other hand, when the traced line graph reaches the end point (S415: Y), the process proceeds to S416.
In S416, the component extraction unit 14 extracts each of the traced line graphs as a component of the line graph, and proceeds to S5.

FIG. 10 is a diagram illustrating an example of a line graph traced from a line graph image by the component extraction unit 14 of the graph image analysis device 1 using a probability map of the line graph.
With reference to FIG. 10, the line graph of the legend A is a solid line, the line graph of the legend B is a broken line, and the line graph of the legend C is a dashed line. Each legend A to C contains two line graphs that intersect each other, the thick line graph plots the values for each year and has a steep waveform, and the thin line graph shows the moving average for multiple years and is smooth. Has a waveform.

As shown in FIG. 10, for example, when two line graphs intersect, in the present embodiment, the line smoothness evaluation function E _smartphone is used to determine in which direction the line is traced from the intersection. Therefore, even if the colors and line types are the same as each other, the thin line graph that intersects with the thick line graph can be distinguished from the thick line graph and traced accurately.

FIG. 11 is a diagram illustrating an example in which the value extraction unit 15 of the graph image analysis device 1 extracts (generates) graph values from a line graph image.
With reference to FIG. 11, the text box detected by scanning the graph image is shown in a rectangular area. Since the value extraction unit 15 scans the entire graph image, in FIG. 11, in addition to the labels of the X-axis axis and the Y-axis axis, the label given to the line graph in the drawing area of the line graph is detected as a text box. .. In the present embodiment, the value extraction unit 15 recognizes the text in the detected text box, and when a numerical value is recognized, calculates the scale of the graph from the recognized numerical value, and the value of the graph corresponding to each label. To generate.

As described above, according to the present embodiment, the graph image analysis device classifies graph images by graph type, calculates a probability map of different graph images for each classified graph type, and calculates the probability. Based on the map, extract the components in the graph image and trace the graph of the extracted components. The graph image analysis device further extracts the values of the components of the extracted graph image, adds the values of the extracted components to the traced graph, and outputs the values.

A different probability map can be calculated for each classified graph type, and each component of the graph can be generically and effectively separated from the graph image by using the calculated probability map.
Therefore, it becomes possible to analyze a wider variety of graph images, and the availability of a large number of graph images scattered on the Web or the like is improved.

<Hardware configuration of graph image analysis device>
FIG. 12 is a diagram showing a non-limiting example of the hardware configuration of the graph image analysis device 1 according to the present embodiment.
The graph image analysis device 1 according to the present embodiment can be implemented on a single or a plurality of computers, mobile devices, or any other processing platform.
Although an example in which the graph image analysis device 1 is mounted on a single computer is shown with reference to FIG. 12, the graph image analysis device 1 according to the present embodiment is used in a computer system including a plurality of computers. May be implemented. A plurality of computers may be connected to each other so as to be able to communicate with each other by a wired or wireless network.

As shown in FIG. 12, the graph image analysis device 1 may include a CPU 121, a ROM 122, a RAM 123, an HDD 124, an input unit 125, a display unit 126, a communication I / F 127, and a system bus 128. .. The graph image analysis device 1 may also include an external memory.
The CPU (Central Processing Unit) 121 comprehensively controls the operation of the graph image analysis device 1, and controls each component (122 to 127) via the system bus 128, which is a data transmission path.
The graph image analysis device 1 may also include a GPU (Graphics Processing Unit). The GPU has a higher computing function than the CPU 121, and by operating a plurality or a large number of GPUs in parallel, it provides higher processing performance, particularly for an image processing application using machine learning as in the present embodiment. do. The GPU typically includes a processor and shared memory. Each processor acquires data from a high-speed shared memory and executes a common program to execute a large amount of the same kind of calculation processing at high speed.

The ROM (Read Only Memory) 122 is a non-volatile memory for storing a control program or the like necessary for the CPU 121 to execute a process. The program may be stored in a non-volatile memory such as an HDD (Hard Disk Drive) 114 or SSD (Solid State Drive) or an external memory such as a removable storage medium (not shown).
The RAM (Random Access Memory) 123 is a volatile memory, and functions as a main memory, a work area, or the like of the CPU 121. That is, the CPU 121 loads a program or the like required from the ROM 122 into the RAM 123 when executing the process, and executes the program or the like to realize various functional operations.

The HDD 124 stores, for example, various data and various information necessary for the CPU 121 to perform processing using a program. Further, the HDD 124 stores, for example, various data and various information obtained by the CPU 121 performing processing using a program or the like.
The input unit 125 is composed of a pointing device such as a keyboard and a mouse.
The display unit 126 is composed of a monitor such as a liquid crystal display (LCD). The display unit 126 provides a GUI (Graphical User Interface) for instructing and inputting various parameters used in the graph image analysis process, communication parameters used in communication with other devices, and the like to the graph image analysis device 1. You can do it.

The communication I / F 127 is an interface that controls communication between the graph image analysis device 1 and the external device.
The communication I / F 127 provides an interface with the network and executes communication with an external device via the network. Image data, various parameters, and the like are transmitted and received to and from an external device via the communication I / F 127. In the present embodiment, the communication I / F 127 may execute communication via a wired LAN (Local Area Network) or a dedicated line conforming to a communication standard such as Ethernet (registered trademark). However, the network that can be used in this embodiment is not limited to this, and may be configured by a wireless network. This wireless network includes a wireless PAN (Personal Area Network) such as Bluetooth®, ZigBee®, UWB (Ultra Wide Band) and the like. Further, it includes a wireless LAN (Local Area Network) such as Wi-Fi (Wi-Filess Fidelity) (registered trademark) and a wireless MAN (Metropolitan Area Network) such as WiMAX (registered trademark). Further, it includes a wireless WAN (Wide Area Network) such as LTE / 3G, 4G, and 5G. The network may connect each device so as to be communicable with each other, and the communication standard, scale, and configuration are not limited to the above.

At least a part of the functions of each element of the graph image analysis device 1 shown in FIG. 1 can be realized by executing a program by the CPU 121. However, at least a part of the functions of each element of the graph image analysis device 1 shown in FIG. 1 may be operated as dedicated hardware. In this case, the dedicated hardware operates based on the control of the CPU 121.

Although specific embodiments have been described above, the embodiments are merely examples and are not intended to limit the scope of the present invention. The devices and methods described herein can be embodied in forms other than those described above. Further, without departing from the scope of the present invention, omissions, substitutions and modifications can be made to the above-described embodiments as appropriate. Such abbreviations, substitutions and modifications are included in the claims and equivalents thereof and fall within the technical scope of the invention.

1 ... Graph image analyzer, 3 ... Input data set, 4 ... Output data set, 11 ... Data acquisition unit, 12 ... Graph classification unit, 13 ... Probability map generation unit, 14 ... Component extraction unit, 15 ... Value extraction unit, 16 ... Output unit, 21 ... Input data, 22 ... Classification network, 23 ... Classification result, 24 ... Probability map generation network, 25 ... Probability map, 26 ... Graph trace result, 27 ... Value detection result, 28 ... Output data , 121 ... CPU, 122 ... ROM, 123 ... RAM, 124 ... HDD, 125 ... Input unit, 126 ... Display unit, 127 ... Communication I / F, 128 ... Bus

Claims (16)

The graph image acquisition unit that acquires the graph image and
A graph classification unit that classifies the graph image acquired by the graph image acquisition unit by graph type, and a graph classification unit.
A probability map generation unit that generates a different type of probability map for each graph type classified by the graph classification unit from the graph image, and a probability map generation unit.
A component extraction unit that extracts components in the graph image based on the probability map generated by the probability map generation unit, and a component extraction unit.
A value extraction unit that extracts the value of the component of the graph image extracted by the component extraction unit, and a value extraction unit.
An information processing apparatus including an output unit that outputs the value of the component extracted by the value extraction unit. The information processing apparatus according to claim 1, wherein the probability map generation unit uses a neural network to generate the probability map of the graph image. The information processing apparatus according to claim 2, wherein the neural network includes different output channels for each graph type. The information processing apparatus according to any one of claims 1 to 3, wherein the probability map generation unit generates a plurality of different types of probability maps for separating different components for each graph type. .. The information processing apparatus according to any one of claims 1 to 4, wherein the component extraction unit extracts the component in the graph image and traces the graph. The information processing apparatus according to claim 5, wherein the output unit adds the value of the component extracted by the value extraction unit to the graph traced by the component extraction unit and outputs the value. .. The component extractor calculates the cost of each pixel of the graph image of the line graph using the first kind of probability map for the line graph, and traces the line graph on the pixel with the minimum cost. The information processing apparatus according to any one of claims 5 or 6, wherein the information processing apparatus is to be used. The information processing apparatus according to claim 7, wherein the component extraction unit calculates the cost of each pixel of the graph image of the line graph by using a dynamic programming method. The information processing apparatus according to claim 7, wherein the component extraction unit calculates the cost based on a function for evaluating the smoothness of lines in the line graph. The component extraction unit according to any one of claims 7 to 9, wherein the component extraction unit calculates the cost based on a function for evaluating the consistency of the line color or line type of the line graph. Information processing device. The component extraction unit detects the start point and the end point of the line graph by stacking the existence probabilities of the line graph in the Y coordinate axis direction at each X coordinate position using the first type of probability map. The information processing apparatus according to any one of claims 7 to 10, wherein the line graph is traced from the detected start point to the end point. 5. The information processing apparatus according to any one of the following items. The information processing apparatus according to claim 12, wherein the component extraction unit uses CCA (Connected Component Analysis) to connect the adjacent pixels of the graph image of the graph. The information processing apparatus according to claim 12, wherein the component extraction unit corrects the extracted component with reference to the probability map of the second type. It is an information processing method executed by an information processing device.
Steps to get a graph image and
Steps to classify the acquired graph images by graph type,
From the graph image, a step of generating a different type of probability map for each of the classified graph types, and
Based on the generated probability map, the steps to extract the components in the graph image and
A step of extracting the value of the component of the extracted graph image, and
An information processing method comprising a step of outputting the value of the extracted component. It is an information processing program for causing a computer to execute information processing, and the program causes the computer to execute information processing.
Graph image acquisition process to acquire graph image and
A graph classification process for classifying the graph image acquired by the graph image acquisition process for each graph type, and a graph classification process.
A probability map generation process that generates a different type of probability map for each graph type classified by the graph classification process from the graph image, and a probability map generation process.
A component extraction process for extracting components in the graph image based on the probability map generated by the probability map generation process, and a component extraction process.
A value extraction process for extracting the value of the component of the graph image extracted by the component extraction process, and a value extraction process for extracting the value of the component.
An information processing program for executing a process including an output process for outputting the value of the component extracted by the value extraction process.

Images