JP2020140450A - Structured data generation method and structured data generation device - Google Patents

Abstract

To provide means for generating information belonging to a handwritten drawing as structured data.SOLUTION: A structured data generation method includes: a steps of extracting text information including position information of a text in a handwritten image and drawing information including position information of a drawing from the handwritten image; a discrimination step of discriminating a type of the handwritten image on the basis of the text information and the drawing information; a step of generating element data obtained by associating the text information with the drawing information according to a positional relation between the text and the drawing on the basis of the extracted position information of the text and the extracted position information of the drawing; and a structured data generation step of generating structured data on the basis of feature information corresponding to the discriminated type of the handwritten image and the generated element data.SELECTED DRAWING: Figure 3

Description

The present invention relates to a structured data generation method and a structured data generation device that generate structured data based on a handwritten image.

The conventional structured data generator recognizes individual elements contained in a figure in which handwritten characters, figures, symbols, etc. are mixed, and even after erroneously determining characters, figures, symbols, etc., new additions are made. It is dynamically converted into appropriate characters, figures, and symbols together with the image to be obtained (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2000-99222

However, in the conventional technique, when a handwritten figure is converted into electronic data by an image reading device or the like, the connection and hierarchical structure of each element (object) are not recognized, and the information contained in the handwritten figure is structured data. There is a problem that it cannot be used as.

An object of the present invention is to solve such a problem, and an object of the present invention is to generate information contained in a handwritten figure as structured data.

Therefore, the present invention is a structured data generation method in a structured data generation device, and is a step of extracting text information including position information of text in the handwritten image and graphic information including position information of a figure from a handwritten image. And the determination step of determining the type of the handwritten image based on the text information and the graphic information, and the text information based on the positional relationship between the text and the graphic based on the position information of the extracted text and the positional information of the graphic. A step of generating element data in which the data is associated with the graphic information, a step of generating structured data based on the feature information corresponding to the determined type of handwritten image, and the generated element data. It is characterized by having.

The present invention in this manner has the effect that the information contained in the handwritten figure can be generated as structured data.

Block diagram showing the configuration of the structured data generation device in the embodiment Explanatory drawing of prediction model in Example Flowchart showing the flow of structured data generation processing in the embodiment Explanatory drawing of handwritten flowchart in Example Explanatory drawing of handwritten line chart in Example Explanatory drawing of text information and graphic information in an Example Explanatory drawing of text information and graphic information in an Example Explanatory drawing of element data in Example Explanatory drawing of element data in Example Flowchart showing the flow of element data creation processing in the embodiment Explanatory drawing of configuration data in an Example Explanatory drawing of structured data in Example Explanatory drawing of configuration data in an Example Explanatory drawing of structured data in Example

Hereinafter, examples of the structured data generation method and the structured data generation apparatus according to the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a configuration of a structured data generation device in an embodiment.

In FIG. 1, the structured data generation device 100 is a computer device or the like that generates structured data based on a handwritten image including handwritten characters (text) and figures, and is an object extraction unit 110 and an image type determination unit 120. It has an element data creation unit 130, a structured data creation unit 140, and a structured image creation unit 150.

Further, the structured data generation device 100 includes a control unit such as a CPU (Central Processing Unit) and a storage unit such as a memory, and the entire operation is performed by the control unit based on the control program (software) stored in the storage unit. Be controlled.

The object extraction unit 110 is connected to a handwritten image acquisition unit (not shown) for acquiring the handwritten image 200, and attribute information of each element (object) from the handwritten image 200 acquired (input) by the handwritten image acquisition unit. Is extracted as text information and graphic information. The object extraction unit 110 extracts text information including position information of text in the handwritten image 200 and graphic information including position information of figures such as a quadrangle, an ellipse, and an arrow from the input handwritten image 200.

The handwritten image acquisition unit acquires a handwritten image 200 drawn on a subject such as paper or a whiteboard using an imaging element such as a camera or an image scanner, or scans a tablet and points on the tablet with a finger, a pen, or the like. A handwritten image 200 traced by a pointing device such as a laser pointer is acquired.

Here, the text information is the attribute information of the character (text) written on the handwritten image 200, and is, for example, various information related to the character and the character such as the size, font, color, and position of the character.

Further, the graphic information is attribute information of the graphic written on the handwritten image 200, and is, for example, various information related to characters such as the shape of the graphic and the size and position of the graphic.

The handwritten image 200 is image data of a handwritten figure (flow chart, line chart (process chart), sequence diagram, state chart diagram, etc.) that is read by an image reading device such as a scanner and generated.

Further, the text information includes coordinates (XY coordinates in the handwritten image 200) representing the positions of the characters in the handwritten image 200 in addition to the characters written in the handwritten image 200. The graphic information includes vector information such as the type of graphic (quadrangle, ellipse, arrow, etc.), coordinates representing the position of the graphic (XY coordinates in the handwritten image 200), and size.

The image type determination unit 120 determines the type of handwritten image (flow chart, line chart (process chart), sequence diagram, state chart) based on the input handwritten image 200 and the text information and graphic information extracted by the object extraction unit 110. (Figures, etc.) are discriminated.

The image type determination unit 120 uses an AI (Artificial Intelligence) prediction model to determine the type of handwritten image based on text information and graphic information.

FIG. 2 is an explanatory diagram of the prediction model in the embodiment. As shown in FIG. 2, in order to be able to determine the type of image, the AI learning process uses a handwritten image and text information and graphic information extracted from the handwritten image as input data, and is a diagram of the handwritten image. AI learning is executed using the type of the image indicated by the above as teacher data, and an AI prediction model is generated.

The AI prediction model obtained by this learning can predict the type of an image by using a handwritten image and text information and graphic information as input data. Therefore, the image type discrimination unit 120 uses the AI prediction model to discriminate the type of the handwritten image.

In this way, the image type determination unit 120 uses the handwritten image and the text information and graphic information extracted from the handwritten image as input data, and uses the handwritten image type as teacher data to perform AI learning. Use a model.

The element data creation unit 130 creates (generates) element data in which the text information and the graphic information are associated with each other based on the positional relationship between the text and the graphic based on the text position information and the graphic position information extracted by the object extraction unit 110. Is what you do.

The element data is information in which the text information arranged in the vicinity and the graphic information are associated with each other based on the position information included in the text information and the position information included in the graphic information. In this embodiment, when the distance between the position of the text and the position of the graphic is equal to or less than the threshold value, the element data is generated by associating the text information with the graphic information.

When the text information and the graphic information cannot be associated with each other, the respective text information and the graphic information constitute one element data.

The structured data creation unit 140 is based on the feature information corresponding to the type of the handwritten image and the element data created by the element data creation unit 130 according to the type of the handwritten image determined by the image type determination unit 120. It creates structured data.

Here, the structured data is information representing the attributes of each element (object) drawn on the handwritten image 200, and is data in which the connection and hierarchical structure of each element (object) are recognized. In this embodiment, the structured data is described using a markup language such as XML (Extension Markup Language), and is used as data that recognizes the connection and hierarchical structure of each element (object) drawn on the handwritten image 200. ..

This structured data can be input to various software such as document editing software, spreadsheet software, project management tools, and UML modeling tools.

Further, the feature information is information that is characteristic of the handwritten image, and is information stored in advance in a storage unit such as a memory corresponding to the type of the handwritten image.

For this feature information, for example, when the type of the handwritten image is "flow chart", the element data of the attribute information is a straight line (Type = Line) or the arrow (Type = Arrow) is "connection data", and the attribute information is a straight line (Type = Line). ) Or the element data other than the arrow (Type = Arrow) is the information as "configuration data".

When the type of the handwritten image is "line table", the element data of the attribute information is a straight line (Type = Line) or an arrow (Type = Arrow) is "configuration data", and the attribute information is an element of an item (Type = Item). The data is "item data", and the element data whose attribute information is date (Type = Date) is "date data".

The structured image creation unit 150 creates an image shaped from the structured data created by the structured data creation unit 140.

The structured data generation device 100 configured in this way generates structured data based on a handwritten image including handwritten characters (text) and figures.

The operation of the above-described configuration will be described.

The structured data generation process performed by the structured data generation device will be described with reference to FIGS. 1 and 2 according to the steps represented by S in the flowchart showing the flow of the structured data generation process in the embodiment of FIG.

S301: The object extraction unit 110 of the structured data generation device 100 recognizes texts (characters) and figures included in the input handwritten image 200, their positions and sizes, and extracts text information and graphic information. For example, the object extraction unit 110 extracts the text information and the graphic information shown in FIG. 6 from the handwritten image 200a of the flowchart shown in FIG. 4, and the text shown in the line table (process chart) shown in FIG. Extract information and graphic information.

S302: The image type determination unit 120 uses the input handwritten image 200 and the AI prediction model shown in FIG. 2 based on the text information and graphic information extracted by the object extraction unit 110 in S301 to type the handwritten image. To determine. (Discrimination step)
For example, the image type discrimination unit 120 discriminates the handwritten image 200a shown in FIG. 4 as a “flow chart” and the handwritten image 200b shown in FIG. 5 as a “line chart”.

S303: The element data creation unit 130 associates the text information extracted by the object extraction unit 110 with the graphic information, and performs an element data creation process for creating the element data.

The element data creation unit 130 associates, for example, a text with a figure surrounding the text, or associates the text with a line or an arrow in the vicinity of the text, and groups the text information and the graphic information with the element data. To create.

The element data creation unit 130 creates the element data shown in FIG. 8 based on the text information and the graphic information shown in FIG. 6, and also creates the element data shown in FIG. 9 based on the text information and the graphic information shown in FIG. To do.

The details of the element data creation process performed by the element data creation unit 130 will be described later.

S304: The structured data creation unit 140 determines a method for creating structured data based on the type of the handwritten image determined by the image type determination unit 120 in S302.

The structured data creation unit 140 performs the processing of S305a when the type of the handwritten image is a flowchart, processes S305b when the type of the handwritten image is a line chart, and S305n when the type of the handwritten image is a sequence diagram. Is performed, and structured data is created from the element data created by the element data creation unit 130.

S305a: When the type of the handwritten image is a flowchart, the structured data creation unit 140 creates structured data based on the element data according to the characteristics of the flowchart, and shifts the processing to S306. (Structured data generation step)
In order to create structured data, the structured data creation unit 140 classifies element data into "connection data" of Type = Line or Type = Arrow and "structured data" having other types.

Next, the structured data creation unit 140 confirms the data closest to the start point position (start point position coordinates) and end point position (end point position coordinates) of the respective connection data. When the nearest neighbor data of the start point position or end point position of the connection data is different connection data, the structured data creation unit 140 sets the start point position or end point position of the corresponding connection data to the start point of another connection data in the nearest neighbor. Change to a position or end position.

After changing to the start point position or end point position of the connection data, the structured data creation unit 140 connects with the configuration data in the vicinity of the start point position or end point position of the connection data.

In this way, the structured data creation unit 140 uses graphic information having information representing a straight line (Line) or an arrow (Arrow) as information of a plurality of connection objects (connection data), and other graphic information and other graphic information. The text information is classified as the information (configuration data) of the configuration object, the position information of the second connection object closest to the first connection object is changed to the position information of the first connection object, and the first Structured data is generated by associating the connection object and the second connection object with the configuration object closest to the first connection object and the second connection object.

FIG. 11 shows an example of the post-connection configuration data created from the element data shown in FIG. As shown in FIG. 11, the structured data creation unit 140 creates a branch tag after creating a branch tag when one configuration data is a start point (<From = ...>) of a plurality of connection data. It is output as the structured data shown in 12.

At that time, the structured data creation unit 140 confirms whether the type of the configuration data is “rhomb” (diamond), and if the type is not “rhomb”, an error is made.

In this way, when one configuration object is associated with a plurality of connection objects and is the starting point of the plurality of connection objects, the structured data creation unit 140 determines that the attribute of the configuration object is not a rhombus. To do. This makes it possible to generate more accurate structured data.

S305b: When the type of the handwritten image is a line table, the structured data creation unit 140 creates structured data based on the element data according to the characteristics of the line table, and shifts the processing to S306. (Structured data generation step)
In order to create structured data, the structured data creation unit 140 uses "structured data" of Type = Line or Type = Arrow, "item data" of Type = Item, and "item data" of Type = Date in order to create structured data. Classify as "date data".

After classifying the element data, the structured data creation unit 140 associates the configuration data with the date data in the vicinity of the X coordinate of the start point and the end point of the configuration data.

In this way, the structured data creation unit 140 uses graphic information having information representing a straight line (Line) or an arrow (Arrow) as information (configuring data) of the constituent object, and text information having information representing an item. Is classified as item object information (item data), and text information having information representing a date is classified as date object information (date data), and the configuration object is associated with the date object closest to the configuration object. Generate structured data.

FIG. 13 shows an example of configuration data associated with date data.

Next, the structured data creation unit 140 associates the configuration data in the vicinity of the Y coordinate of the item data and outputs it as the structured data shown in FIG.

S305n: When the type of the handwritten image is a sequence diagram, the structured data creation unit 140 creates structured data based on the element data according to the characteristics of the sequence diagram, and shifts the processing to S306. (Structured data generation step)
The detailed description of the process of creating the structured data of the sequence diagram will be omitted.

S306: The structured data creation unit 140 outputs the structured data created in S305a to S305n as a file.

S307: The structured image creation unit 150 shapes a handwritten image based on the structured data created by the structured data creation unit 140, and the structured image data (image) that makes it easy to recognize the connection and hierarchical structure of each object. Is created and output, and this process ends.

Next, the element data creation process performed by the structured data generation device will be described with reference to FIGS. 1 and 2 according to the steps represented by S in the flowchart showing the flow of the element data creation process in the embodiment of FIG. It should be noted that this element data creation process is a process performed by the element data creation unit 130 in S303 of FIG.

S401: The element data creation unit 130 searches for the text information (for example, the text information and the graphic information shown in FIGS. 6 and 7) extracted by the object extraction unit 110, and when the data related to the date is detected, the type of the text information Is converted from "text" to "date". For example, since the information of ID = 2 in FIG. 7 is data related to the date (“3 / B”), it is converted from Type = Text to Type = Date. The element data creation unit 130 classifies the text information into Type = Text and Type = Date, and registers the classified text information in List 1.

S402: The element data creation unit 130 registers information other than text information including date information (information other than Type = Text and Type = Date), that is, graphic information in Listing 2.

S403: The element data creation unit 130 extracts one graphic information from the list 2 and deletes the extracted graphic information from the list 2.

S404: The element data creation unit 130 determines whether or not the extracted graphic information is a line (Type = Line) or an arrow (Type = Arrow), and if the extracted graphic information is a line or an arrow, the process shifts to S405. Then, if the extracted graphic information is not a line or an arrow, the process shifts to S406.

S405: When the extracted graphic information is a line or an arrow, the element data creation unit 130 determines from the information registered in the list 1 whether or not there is (exists) text information in the vicinity of the graphic information, and the graphic is If it is determined that there is text information in the vicinity of the information, the process shifts to S407, and if it is determined that there is no text information in the vicinity of the graphic information, the process shifts to S409.

The element data creation unit 130 calculates a distance using lines or arrows and position information of text information, and if the distance is equal to or less than a specific threshold value, determines that there is text information in the vicinity of the graphic information, and determines that there is text information in the vicinity of the graphic information. If it is larger than, it is determined that there is no text information in the vicinity of the graphic information.

For example, it is determined that the text information (print / scan) with ID = 3 and the graphic information (arrow) with ID = 4 shown in FIG. 7 are in the vicinity.

S406: When the extracted graphic information is not a line or an arrow, the element data creation unit 130 determines whether or not there is (exists) text information in the graphic of the extracted graphic information based on the information registered in Listing 1. When the determination is made and it is determined that there is text information in the figure, the process shifts to S407, and when it is determined that there is no text information in the figure, the process shifts to S409.

For example, it is determined that the text information (start) of ID = 3 is included in the graphic represented by the graphic information of ID = 1 shown in FIG.

S407: When there is text information in the vicinity of the graphic information of the line or arrow, or when there is text information in the graphic of the graphic information, the element data creation unit 130 associates the text information with the graphic information existing in the neighborhood. Create element data.

For example, the information of ID = 1 and ID = 3 in FIG. 6 and the information of ID = 3 and ID = 4 in FIG. 7 are related in this step and are created as element data including text information and graphic information.

S408: The element data creation unit 130 deletes the text information for which the element data is created in association with the graphic information from the list 1, and shifts the process to S410.

S409: The element data creation unit 130 creates element data only with the graphic information when there is no text information in the vicinity of the graphic information of the line or arrow, or when there is no text information in the graphic of the graphic information.

S410: The element data creation unit 130 determines whether or not the graphic information registered in the list 2 is empty (whether or not the graphic information is exhausted), and if it is determined that the graphic information is not empty, the process is sent to S403. The process is returned and the processes of S403 to S409 are repeated, and when it is determined that the process is empty, the process is shifted to S411.

S411: When the element data creation unit 130 determines that the graphic information registered in the list 2 is empty, whether or not the text information registered in the list 1 is empty (whether or not the text information is exhausted). ) Is determined, and if it is determined that the process is not empty, the process shifts to S412, and if it is determined that the process is empty, the present process is terminated.

S412: When the text information registered in the list 1 is not empty, the element data creation unit 130 sets the text information as Type = Item, creates element data only with the text information, and ends this process.

The element data creation unit 130 creates the element data of FIGS. 8 and 9, respectively, based on the text information and the graphic information of FIGS. 6 and 7, respectively, by performing the processes of S401 to S412.

As described above, in the present embodiment, in the structured data generation device 100, the object extraction unit 110 extracts the text information including the position information of the text in the handwritten image and the graphic information including the position information of the graphic from the handwritten image ( S301 in FIG. 3), the image type determination unit 120 determines the type of the handwritten image based on the text information and the graphic information (S302 in FIG. 3), and the element data creation unit 130 determines the type of the text extracted by the object extraction unit 110. Based on the position information and the position information of the figure, the element data that associates the text information and the figure information with the positional relationship of the text and the figure is generated (S303 in FIG. 3), and the structured data creation unit 140 sets the type of the handwritten image. By generating structured data based on the corresponding feature information and the generated element data (S305a, S305b, ..., S305n in FIG. 3), the information possessed by the handwritten figure is structured. It can be generated as data (structured data).

Structured data created from hand-drawn diagrams recognizes the data structure (data hierarchy and connection relationships), so the created structured data can be directly input to various software, and in various software. It is possible to reduce the work of inputting setting information by the user.

For example, when structured data is generated from a handwritten flowchart, it is possible to create an electronic data flowchart in which the connection between each object is clear and easy to see without requiring the user to input detailed setting information. Will be.

In addition, when structured data is generated from a handwritten line chart, the generated structured data can be imported into the Gantt chart of the project management tool without the need for the user to input detailed setting information. become.

Further, when structured data is generated from a handwritten sequence diagram, state chart diagram, etc., it becomes possible to create a class template through a UML (Unified Modeling Language) modeling tool.

In addition, when structured data generated from a hand-created diagram is imported into electronic document software such as document editing software or spreadsheet software, the electronic document becomes a graphic image that is easy for the reader to see and is not misleading. Can be converted.

As described above, in the present embodiment, the effect that the information contained in the handwritten figure can be generated as structured data can be obtained.

Therefore, the structured data created from the handwritten figure can be directly input to various software, and it is possible to reduce the work of inputting the setting information by the user in various software.

In this embodiment, the example of creating structured data from a handwritten flowchart and a handwritten line chart has been described, but the present invention is not limited to this, and other handwritten figures and characters such as handwritten sequence diagrams and state chart diagrams are used. Structured data may be created from the diagram having.

100 Structured data generator 110 Object extraction unit 120 Image type discrimination unit 130 Element data creation unit 140 Structured data creation unit 150 Structured image creation unit 200 Handwritten image

Claims (8)

It is a structured data generation method in a structured data generation device.
A step of extracting text information including the position information of the text in the handwritten image and graphic information including the position information of the graphic from the handwritten image, and
A determination step for determining the type of the handwritten image based on the text information and the graphic information, and
A step of generating element data in which text information and graphic information are associated with each other based on the positional relationship between the text and the graphic based on the extracted text position information and the graphic position information.
A structured data generation step that generates structured data based on the feature information corresponding to the determined handwritten image type and the generated element data.
A structured data generation method characterized by having. In the structured data generation method according to claim 1,
A structured data generation method, wherein the element data is generated by associating the text information with the graphic information when the distance between the position of the text and the position of the graphic is equal to or less than a threshold value. In the structured data generation method according to claim 2,
When the type of the handwritten image is determined to be a flowchart in the determination step,
In the structured data generation step, the element data is classified into graphic information having information representing a straight line or an arrow as information of a plurality of connected objects, and other graphic information and text information as information of constituent objects. The position information of the second connection object closest to the connection object is changed to the position information of the first connection object, and the first connection object and the second connection object are connected to the first connection. A method for generating structured data, characterized in that the structured data is generated by associating the object with the constituent object closest to the second connection object. In the structured data generation method according to claim 3,
A structure characterized in that when one configuration object is associated with a plurality of connection objects and serves as a starting point of the plurality of connection objects, it is determined to be abnormal when the attribute of the configuration object is not a rhombus. Data generation method. In the structured data generation method according to claim 2,
When the type of the handwritten image is determined to be a process chart in the determination step,
In the structured data generation step, the element data has graphic information having information representing a straight line or an arrow as information of a constituent object, text information having information representing items as information of an item object, and information representing a date. A structured data generation method characterized in that text information is classified as information of a date object, and the structured data is generated by associating the constituent object with the date object closest to the constituent object. In the structured data generation method according to any one of claims 1 to 5.
In the determination step,
It is characterized in that a prediction model generated as a result of executing AI learning is used by using the handwritten image and text information and graphic information extracted from the handwritten image as input data and using the type of the handwritten image as teacher data. Structured data generation method. In the structured data generation method according to any one of claims 1 to 6,
A structured data generation method comprising a step of generating structured image data in which the connection of each object and the hierarchical structure are easily recognized from the structured data generated in the structured data generation step. An object extraction unit that extracts text information including the position information of the text in the handwritten image and graphic information including the position information of the graphic from the input handwritten image, and an object extraction unit.
An image type determination unit that determines the type of handwritten image based on the text information and the graphic information,
An element data creation unit that generates element data in which text information and graphic information are associated with each other based on the positional relationship between the text and the graphic based on the text position information and the graphic position information extracted by the object extraction unit.
A structured data creation unit that generates structured data based on feature information corresponding to the type of handwritten image determined by the image type determination unit and element data generated by the element data creation unit.
A structured data generator characterized by having.

Images