KR20230154600A - Method and device for generating training data of model for table recognition - Google Patents

Abstract

A method for generating training data for a model for table recognition performed by a computing device is disclosed. The method includes: generating a table containing a plurality of zones; establishing a first header zone and a first content zone on the plurality of zones; generating at least one of a horizontal header, a vertical header, and a merge header in the first header area; merging or splitting cells on the first content area, or establishing a second header area and a second content area; and determining a table structure based at least on the first header section and the first content section; may include.

Description

Method and device for generating training data of a model for table recognition {METHOD AND DEVICE FOR GENERATING TRAINING DATA OF MODEL FOR TABLE RECOGNITION}

The present invention relates to computer technology, and more specifically to a method and device for generating training data for a model for table recognition.

Table Structure Recognition (TSR) is a technology that generates data about the table structure in a machine-readable format (html, spreadsheet, etc.) by recognizing the table structure that appears in image files such as jpg, png, etc. For commonly used tables, it is necessary to identify the actual size of the cell. For example, the actual size of the cells must be identified in order to convert tables that require the user to write or enter, such as documents and various contracts prepared in government offices, into digital data. In particular, in order to accurately recognize the table structure shown in the image, the exact size of each cell must be identified.

In order to use a machine learning model for table recognition, data about various types of tables must be secured. The table may have a complex structure, such as multiple cells being merged or many empty cells. In addition, most of the table data that can be collected existing has problems that make it difficult to utilize, such as containing personal information.

Therefore, there is a demand for technology that synthesizes various types of table data to learn a model for table recognition.

This disclosure was made in response to the above-described background technology and relates to a method and device for generating training data for a model for table recognition.

A method of generating training data for a model for table recognition performed by a computing device for realizing the above-described tasks is disclosed. The method includes: generating a table containing a plurality of zones; establishing a first header zone and a first content zone on the plurality of zones; generating at least one of a horizontal header, a vertical header, and a merge header in the first header area; merging or splitting cells on the first content area, or establishing a second header area and a second content area; and determining a table structure based at least on the first header section and the first content section; may include.

Alternatively, generating at least one of a horizontal header, a vertical header, and a merge header on the first header section includes: on the first row of the R rows for generating a horizontal header among the first header section, generating a first row header including at least one section; And on the nth row among the R rows, each zone corresponding to at least one zone included in the n-1th row header of the preceding n-1th row is divided into one or more zones to create an nth row header. generating step; may include.

Alternatively, generating at least one of a horizontal header, a vertical header, and a merge header on the first header section may include: generating at least one of adjacent sections of the same column width on at least two consecutive columns of the C columns; merging parts; may further include.

Alternatively, the step of generating at least one of a horizontal header, a vertical header, and a merge header on the first header section: on a first column of C columns for which a vertical header is to be created among the first header section; , generating a first column header including at least one row section; And on the m-th column of the C columns, generate the m-th row header by dividing each zone corresponding to at least one zone included in the m-1-th column header of the preceding m-1-th column into one or more zones. It may include;

Alternatively, generating at least one of a horizontal header, a vertical header, and a merge header on the first header section may include: generating at least one of adjacent sections of the same column width on at least two consecutive columns of the C columns; merging parts; may further include.

Alternatively, generating at least one of a horizontal header, a vertical header, and a merged header on the first header section includes: merging overlapping sections of rows 1 to R and columns 1 to C of the first header section. It may include creating a merge-type header section by doing so.

Alternatively, merging or splitting cells on the first content area, or establishing a second header area and a second content area: corresponding to the generated horizontal header, vertical header, or merged header dividing the first content area so as to; may include.

Alternatively, merging or splitting cells on the first content area, or establishing a second header area and a second content area, may include: merging at least a portion of the divided first content area; dividing at least a portion of the divided first content zones into at least one zone; or setting a second header area and a second content area in at least some of the divided first content areas; may further include.

Alternatively, the method may include: setting pixel values of rows and columns of the determined table structure; It may further include.

Alternatively, the method may include: setting text on at least some of the cells included in the determined table structure; may further include.

Alternatively, the method may include: setting a style on at least some of the cells included in the determined table structure; may further include.

Alternatively, the style may include at least one of a border style, text font, or background color.

A computer program stored in a computer-readable medium for realizing the above-described object is disclosed. The computer program includes instructions for causing one or more processors to perform a method of generating training data for a table recognition model, the method comprising: generating a table including a plurality of regions; establishing a first header zone and a first content zone on the plurality of zones; generating at least one of a horizontal header, a vertical header, and a merge header in the first header area; merging or splitting cells on the first content area, or establishing a second header area and a second content area; and determining a table structure based on at least the first header section and the first content section.

A computing device that performs a method of generating training data for a table recognition model for realizing the above-described tasks is disclosed. The computing device includes: memory including computer-executable components; a processor executing the following computer-executable components stored in memory; wherein the processor generates a table including a plurality of zones, sets a first header zone and a first content zone on the plurality of zones, and sets a horizontal header and a vertical header on the first header zone. Create at least one of a header, a merged header, merge or split cells on the first content area, or set a second header area and a second content area, and at least the first header area and the second content area. 1 The table structure can be determined based on the content area.

The present disclosure can provide a method and device for generating training data for a model for table recognition.

FIG. 1 is a block diagram of a computing device that performs an operation to provide a method of generating training data for table recognition according to some embodiments of the present disclosure.
Figure 2 is a schematic diagram showing network functions according to some embodiments of the present disclosure.
Figure 3 is a flowchart of a method for generating training data for a model for table recognition according to some embodiments of the present disclosure.
4 is an example table structure generated by a method for generating training data for a model for table recognition according to some embodiments of the present disclosure.
Figure 5 is a diagram for explaining an operation of creating a horizontal header according to some embodiments of the present disclosure.
FIG. 6 is a diagram for explaining an operation of generating a merged header according to some embodiments of the present disclosure.
7 is a block diagram of a computing device according to some embodiments of the present disclosure.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the disclosure. However, it is clear that these embodiments may be practiced without these specific descriptions.

As used herein, the terms “component,” “module,” “system,” and the like refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or an implementation of software. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and the computing device can be a component. One or more components may reside within a processor and/or thread of execution. A component may be localized within one computer. A component may be distributed between two or more computers. Additionally, these components can execute from various computer-readable media having various data structures stored thereon. Components can transmit signals, for example, with one or more data packets (e.g., data and/or signals from one component interacting with other components in a local system, a distributed system, to other systems and over a network such as the Internet). Depending on the data being transmitted, they may communicate through local and/or remote processes.

Additionally, the term “or” is intended to mean an inclusive “or” and not an exclusive “or.” That is, unless otherwise specified or clear from context, “X utilizes A or B” is intended to mean one of the natural implicit substitutions. That is, either X uses A; X uses B; Or, if X uses both A and B, “X uses A or B” can apply to either of these cases. Additionally, the term “and/or” as used herein should be understood to refer to and include all possible combinations of one or more of the related listed items.

Additionally, the terms “comprise” and/or “comprising” should be understood to mean that the corresponding feature and/or element is present. However, the terms “comprise” and/or “comprising” should be understood as not excluding the presence or addition of one or more other features, elements and/or groups thereof. Additionally, unless otherwise specified or the context is clear to indicate a singular form, the singular terms herein and in the claims should generally be construed to mean “one or more.”

And, the term “at least one of A or B” should be interpreted to mean “a case containing only A,” “a case containing only B,” and “a case of combining A and B.”

Those skilled in the art will additionally recognize that the various illustrative logical blocks, components, modules, circuits, means, logic, and algorithm steps described in connection with the embodiments disclosed herein may be implemented using electronic hardware, computer software, or a combination of both. It must be recognized that it can be implemented with To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or software will depend on the specific application and design constraints imposed on the overall system. A skilled technician can implement the described functionality in a variety of ways for each specific application. However, such implementation decisions should not be construed as causing a departure from the scope of the present disclosure.

The description of the presented embodiments is provided to enable anyone skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the disclosure. Therefore, the present invention is not limited to the embodiments presented herein. The present invention is to be interpreted in the broadest scope consistent with the principles and novel features presented herein.

1 is a block diagram of a computing device for performing a method of generating training data for table recognition according to some embodiments of the present disclosure.

As shown in FIG. 1, the computing device 100 may include a processor 110, a memory 130, and a network unit 150. The configuration of the computing device 100 shown in FIG. 1 is only a simplified example. In some embodiments of the present disclosure, computing device 100 may include different components for performing the computing environment of computing device 100, and only some of the disclosed components may configure computing device 100.

The processor 110 may be composed of one or more cores, and may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. unit) may include a processor for data analysis and processing, and deep learning. The processor 110 may read the computer program stored in the memory 130 and perform data conversion, calculation, generation, etc. for a method of generating training data for a model for table recognition according to some embodiments of the present disclosure. . For example, the processor 110 may perform steps to perform a method of generating training data for a model for table recognition, which will be described below. Additionally, according to some embodiments of the present disclosure, the processor 110 may perform an operation for learning a neural network using training data generated by a method of generating training data for table recognition. The processor 110 is used for learning neural networks, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. Calculations can be performed. At least one of the CPU, GPGPU, and TPU of the processor 110 may process operations for generating training data for a model for table recognition. For example, the CPU and GPGPU can jointly process calculations for generating training data for a model for table recognition. In addition, in some embodiments of the present disclosure, data conversion, operation, generation, learning of network functions, and data using network functions for a method of generating training data for a model for table recognition by using processors of a plurality of computing devices together Classification can be handled. Additionally, a computer program executed on a computing device according to some embodiments of the present disclosure may be a CPU, GPGPU, or TPU executable program.

According to some embodiments of the present disclosure, the memory 130 may store any type of information generated or determined by the processor 110 and any type of information received by the network unit 150. For example, the memory 130 may store data (eg, table structure data) generated in the process of performing a method of generating training data for a model for table recognition by the processor 110. Additionally, the memory 130 may store data received externally during the process of performing a method of generating training data for a model for table recognition by the processor 110. However, the present disclosure is not limited to this, and the memory 130 may store various information for performing a method of generating training data for a model for table recognition according to some embodiments of the present disclosure.

According to some embodiments of the present disclosure, the memory 130 may be a flash memory type, hard disk type, multimedia card micro type, or card type memory (e.g. (e.g. SD or -Only Memory), and may include at least one type of storage medium among magnetic memory, magnetic disk, and optical disk. The computing device 100 may operate in connection with web storage that performs a storage function of the memory 130 on the Internet. The description of the memory described above is merely an example, and the present disclosure is not limited thereto.

The network unit 150 according to some embodiments of the present disclosure may use any type of known wired or wireless communication system.

The network unit 150 can transmit and receive information processed by the processor 110, a user interface, etc. through communication with other terminals. For example, the network unit 150 may provide a user interface generated by the processor 110 to a client (e.g. user terminal). Additionally, the network unit 150 may receive external input from a user authorized as a client and transmit it to the processor 110. At this time, the processor 110 may process operations such as output, modification, change, and addition of information provided through the user interface based on the user's external input received from the network unit 150.

For example, the network unit 150 may transmit and receive various information to perform a method of generating training data for a model for table recognition according to some embodiments of the present disclosure. For example, the network unit 150 may receive a table image used to generate training data from a database. Additionally, the network unit 150 may externally transmit some data generated in the process of performing the method of generating learning data for a model for table recognition described below to be stored in a database.

Meanwhile, the computing device 100 according to some embodiments of the present disclosure is a computing system that transmits and receives information through communication with a client and may include a server. At this time, the client may be any type of terminal that can access the server. For example, the computing device 100, which is a server, may receive a query from a user terminal and generate a single information processing result corresponding to the query. In this case, the computing device 100, which is a server, may provide a user interface including processing results to the user terminal. At this time, the user terminal can output the user interface received from the computing device 100, which is a server, and receive or process information through interaction with the user.

In an additional embodiment, the computing device 100 may include any type of terminal that receives data resources generated by an arbitrary server and performs additional information processing.

2 is a schematic diagram showing a network function according to some embodiments of the present disclosure.

Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. A neural network can generally consist of a set of interconnected computational units, which can be referred to as nodes. These nodes may also be referred to as neurons. A neural network consists of at least one node. Nodes (or neurons) that make up neural networks may be interconnected by one or more links.

Within a neural network, one or more nodes connected through a link may form a relative input node and output node relationship. The concepts of input node and output node are relative, and any node in an output node relationship with one node may be in an input node relationship with another node, and vice versa. As described above, input node to output node relationships can be created around links. One or more output nodes can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of the data of the output node may be determined based on the data input to the input node. Here, the link connecting the input node and the output node may have a weight. Weights may be variable and may be varied by the user or algorithm in order for the neural network to perform the desired function. For example, when one or more input nodes are connected to one output node by respective links, the output node is set to the values input to the input nodes connected to the output node and the links corresponding to each input node. The output node value can be determined based on the weight.

As described above, in a neural network, one or more nodes are interconnected through one or more links to form an input node and output node relationship within the neural network. The characteristics of the neural network can be determined according to the number of nodes and links within the neural network, the correlation between the nodes and links, and the value of the weight assigned to each link. For example, if the same number of nodes and links exist and two neural networks with different weight values of the links exist, the two neural networks may be recognized as different from each other.

A neural network may consist of a set of one or more nodes. A subset of nodes that make up a neural network can form a layer. Some of the nodes constituting the neural network may form one layer based on the distances from the first input node. For example, a set of nodes with a distance n from the initial input node may constitute n layers. The distance from the initial input node can be defined by the minimum number of links that must be passed to reach the node from the initial input node. However, this definition of a layer is arbitrary for explanation purposes, and the order of a layer within a neural network may be defined in a different way than described above. For example, a layer of nodes may be defined by distance from the final output node.

The initial input node may refer to one or more nodes in the neural network through which data is directly input without going through links in relationships with other nodes. Alternatively, in a neural network network, in the relationship between nodes based on links, it may mean nodes that do not have other input nodes connected by links. Similarly, the final output node may refer to one or more nodes that do not have an output node in their relationship with other nodes among the nodes in the neural network. Additionally, hidden nodes may refer to nodes constituting a neural network other than the first input node and the last output node.

The neural network according to an embodiment of the present disclosure is a neural network in which the number of nodes in the input layer may be the same as the number of nodes in the output layer, and the number of nodes decreases and then increases again as it progresses from the input layer to the hidden layer. You can. In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer may be less than the number of nodes in the output layer, and the number of nodes decreases as it progresses from the input layer to the hidden layer. there is. In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer may be greater than the number of nodes in the output layer, and the number of nodes increases as it progresses from the input layer to the hidden layer. You can. A neural network according to another embodiment of the present disclosure may be a neural network that is a combination of the above-described neural networks.

A deep neural network (DNN) may refer to a neural network that includes multiple hidden layers in addition to the input layer and output layer. Deep neural networks allow you to identify latent structures in data. In other words, it is possible to identify the potential structure of a photo, text, video, voice, or music (e.g., what object is in the photo, what the content and emotion of the text are, what the content and emotion of the voice are, etc.) . Deep neural networks include convolutional neural networks (CNN), recurrent neural networks (RNN), auto encoders, generative adversarial networks (GAN), and restricted Boltzmann machines (RBM). machine), deep belief network (DBN), Q network, U network, Siamese network, Generative Adversarial Network (GAN), transformer, etc. The description of the deep neural network described above is only an example and the present disclosure is not limited thereto.

A neural network may be trained in at least one of supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Learning of a neural network may be a process of applying knowledge for the neural network to perform a specific operation to the neural network.

Neural networks can be trained to minimize output errors. In neural network learning, learning data is repeatedly input into the neural network, the output of the neural network and the error of the target for the learning data are calculated, and the error of the neural network is transferred from the output layer of the neural network to the input layer in the direction of reducing the error. This is the process of updating the weight of each node in the neural network through backpropagation. In the case of teacher learning, learning data in which the correct answer is labeled in each learning data is used (i.e., labeled learning data), and in the case of non-teacher learning, the correct answer may not be labeled in each learning data. That is, for example, in the case of teacher learning regarding data classification, the learning data may be data in which each learning data is labeled with a category. Labeled training data is input to the neural network, and the error can be calculated by comparing the output (category) of the neural network with the label of the training data. As another example, in the case of non-teachable learning for data classification, the error can be calculated by comparing the input training data with the neural network output. The calculated error is backpropagated in the reverse direction (i.e., from the output layer to the input layer) in the neural network, and the connection weight of each node in each layer of the neural network can be updated according to backpropagation. The amount of change in the connection weight of each updated node may be determined according to the learning rate. The neural network's calculation of input data and backpropagation of errors can constitute a learning cycle (epoch). The learning rate may be applied differently depending on the number of repetitions of the learning cycle of the neural network. For example, in the early stages of neural network training, a high learning rate can be used to increase efficiency by allowing the neural network to quickly achieve a certain level of performance, and in the later stages of training, a low learning rate can be used to increase accuracy.

Neural networks can use various data augmentation methods to increase the amount of training data for training. For example, data augmentation can be performed through two-dimensional transformations such as rotation, scale, shearing, reflection, and translation. Additionally, data augmentation can be performed using noise insertion, color, brightness modification, etc.

In the learning of neural networks, the training data can generally be a subset of real data (i.e., the data to be processed using the learned neural network), and thus the error for the training data is reduced, but the error for the real data is reduced. There may be an incremental learning cycle. Overfitting is a phenomenon in which errors in actual data increase due to excessive learning on training data. Overfitting can cause errors in machine learning algorithms to increase. To prevent such overfitting, various optimization methods can be used. To prevent overfitting, methods such as increasing the learning data, regularization, dropout to disable some of the network nodes during the learning process, and use of a batch normalization layer can be applied. You can.

3 is a flowchart of a method for generating training data for table recognition according to some embodiments of the present disclosure. 4 is an example table structure generated by a method for generating training data for table recognition according to some embodiments of the present disclosure. Figure 5 is a diagram for explaining an operation of creating a horizontal header according to some embodiments of the present disclosure. FIG. 6 is a diagram for explaining an operation of generating a merged header according to some embodiments of the present disclosure.

Below, with reference to FIGS. 3 to 6 , exemplary embodiments of a method and apparatus for generating training data for a model for table recognition according to some embodiments of the present disclosure are described.

Data sets used in existing table structure recognition (TSR) often do not include pixel information. Therefore, the table recognition model learned from this data set was unable to estimate the exact size of each cell. Additionally, most of the datasets were taken from papers. The table structures included in these datasets mainly have a simple header-content form. This disclosure proposes a method for synthesizing arbitrary table structures to simulate the form of tables used in reality. Tables used in reality may have the following characteristics. For example, merging of cells may occur in the middle of a table (e.g., see bottom left of FIG. 4). As another example, as shown in FIG. 4, a header area may exist in the middle of the table. As another example, as shown in FIG. 4, multiple zones may exist within one table. Additionally, multiple zones may have different styles. The present disclosure can provide a method and device for generating learning data to train a model that can recognize these various types of tables. Specifically, the training data for training a model for table recognition may include a table image, pixel value information of horizontal and vertical lines defining cells related to the table image, merge information between each cell, and text information within each cell. You can. The method and device of the present disclosure may be particularly useful for synthesizing the table structure required to generate pixel value information of horizontal and vertical lines and merge information between each cell.

According to some embodiments of the present disclosure, a method of generating training data for a model for table recognition may include generating a table including a plurality of zones (s100). For example, the plurality of zones may include N x M zones. Here, the integers N and M may be determined randomly or may be determined as predetermined numbers according to user input, etc.

According to some embodiments of the present disclosure, a method of generating training data for a model for table recognition may include setting a first header area and a first content area in a plurality of areas (s200).

A table can contain a header section and a content section. For example, a horizontal header, a vertical header, and/or a merged header can be created in the header section. In some examples, a horizontal header may occupy one or more rows of a table. And a horizontal header can occupy all columns of a given section for the rows it occupies. In some examples, a vertical header may occupy one or more columns of a table. And a vertical header can occupy all columns of a given section for the rows it occupies. A merged header can be a header that contains both a horizontal header and a vertical header.

Typically, horizontal headers can be placed in the top row of a table. Additionally, the vertical header can be located at the left end of the table. The merged header can be located in the form of an 'r' at the top and left end of the table. Considering this header format, in some examples, the first header section can be set in any number of rows at the top of the table to create a horizontal header. As another example, the first header section can be set in any number of columns at the left end of the table to create a vertical header. As another example, the first header section can be set in any number of rows at the top of the table and any number of columns at the left end of the table to create a merged header. Also, the first content area may be set in an area excluding the first header area among the plurality of areas. However, it is not limited to this, and the first header area and first content area can be set in various ways.

According to some embodiments of the present disclosure, a method of generating training data for a model for table recognition includes generating at least one of a horizontal header, a vertical header, and a merge header in a first header area (s300). can do.

When the first header area is set, at least one of a horizontal header, a vertical header, and a merge header may be created in the first header area. As described above, a header of a suitable type may be installed depending on the shape of the first header area, but is not limited to this.

First, with reference to Figure 5, an example embodiment is described in which a horizontal header 100 is created on a first header section. A first row header including at least one area may be generated on the first row among the R rows to generate the horizontal header 100 among the first header areas. The number of zones (number of cells) included in the first row header may be predetermined or determined randomly. Referring to FIG. 5, a state in which a first row header including two zones (zone '1' and zone '2') is created on the first row among the three rows in FIG. 5 is shown.

After the first row header is generated on the first row, the second row header through the R-th row header may be generated on the second row through the R-th row. In some examples, for the nth row of the R rows, the nth row Headers can be created. For example, referring to Figure 5, the second row header is configured to, on the second row, (i) the zone corresponding to zone '1' included in the preceding first row header into zone '1-1' and Split into zone '1-2' (ii) It can be created by dividing the zone corresponding to zone '2' included in the preceding first header into zone '2-1' and zone 2-2'. Similarly, the third row header divides on the third row (i) the zone corresponding to zone '1-1' contained in the preceding second row header into zone '1-1-1' and zone '1-1'; Split into '-2' (ii) Split the zone corresponding to zone '1-2' contained in the preceding second row header into zone '1-2-1' and zone '1-2-2' (iii) Splitting the zone corresponding to zone '2-1' included in the preceding second row header into zone '2-1-1' and zone '2-1-2' (iv) included in the preceding second row header It can be created by dividing the zone corresponding to the zone '2-2' into one zone '2-2'. In the above example, the number of divided zones is 1 or 2, but the number of divided zones may vary.

When it is determined how the sections of the horizontal header 100 are divided, the number of rows to be used across multiple rows can be determined. For example, in at least two consecutive rows among the R rows, at least some of adjacent areas with the same row width may be merged. Referring to FIG. 5, zone '2-2' of the second row and zone '2-2' of the third row may have the same row width. And, the zone '2-2' in the second row and the zone '2-2' in the third row can be merged with each other so that one header occupies two rows. The areas to be merged can be randomly determined so that various types of tables can be created.

Similarly, the vertical header 200 may be created in a similar manner to the horizontal header 100. For example, a first column header including at least one section may be created on the first column among the C columns to generate the vertical header 200 among the first header sections. The number of zones (number of cells) included in the first column header may be predetermined or determined randomly. After the first column header is created on the first column, the second column header through the C-th column can be created on the second column through the C-th column. In some examples, for the m-th column of C columns, the m-th column header is created by dividing each zone corresponding to at least one zone contained in the m-1th row header of the preceding m-1th column into one or more zones. can be created. Here, the number of divided areas may vary.

6, an example embodiment is described in which a merged header 300 is created on a first header section. First, R rows for creating the horizontal header 100 and C columns for creating the vertical header 200 can be determined. Referring to Figure 6, the first header section may include three rows at the top of the table and two columns at the left end of the table. In this case, the merged header section 300 may be created by merging the overlapping sections of rows 1 to R and columns 1 to C of the first header section. Referring to FIG. 6, a merged header section 300 may be created by merging overlapping sections of rows 1 to 3 and columns 1 to 2. Also, the horizontal header 100 or the vertical header 200 may be created in the first header area rather than the merged header area 300.

When the header creation operation is completed on the first header area, the first content area may be divided to correspond to the created horizontal header, vertical header, or merged header. For example, the rows of the first content area may be divided to correspond to the area shape (cell shape) of the bottom row of the horizontal header. Additionally, the columns of the first content zone may be divided to correspond to the zone shape (cell shape) of the rightmost column of the vertical header.

Additionally, the steps described above may be performed recursively on the divided first content area. For example, a second header area and a second content area may be set in at least part of the divided first content area. The method in which the header is generated on the second header section may be the same as the method in which the header is generated on the first header section described above. Additionally, a third header area and a third content area may be set in at least part of the second content area. As another example, at least some of the divided first content areas may be merged. As another example, at least a portion of the divided first content zone may be divided into at least one zone.

According to some embodiments of the present disclosure, a method of generating training data for a model for table recognition may include determining a table structure based on at least the first header section and the first content section.

By recursively performing the steps described above, various types of table structures in which cells are arranged can be created. In some examples, when the structure of cells on a first header section and a first content section is determined, a table structure may be determined based on the first header section and the first content section. In another example, when the structure of the cells on the second header section and the second content section is determined, one table based on the second header section and the second content section together with the first header section and the first content section The structure may be further determined.

According to some embodiments of the present disclosure, a method of generating training data for a model for table recognition may include setting pixel values of rows and columns of the determined table structure. Here, so that the model learned from the training data can accurately predict the size of each cell, the pixel values of the rows and columns can be set in various ways within the range of a general table format.

According to some embodiments of the present disclosure, a method of generating training data for a model for table recognition may further include setting text on at least some of the cells included in the determined table structure. Here, the text may contain letters or numbers from various languages. Additionally, the text may include randomly determined content.

According to some embodiments of the present disclosure, a method of generating training data for a model for table recognition may include setting a style on at least some of the cells included in the determined table structure. Here, the style may include at least one of a border style, text font, or background color. Styles can also be set randomly or in various ways according to presets.

Information about merging of cells created according to the above-described steps, pixel values of rows and columns, text, style, etc. can be stored as table structure data. Table structure data can be used as learning data along with the corresponding table image. Table structure data generated according to the method and apparatus of the present disclosure may have a complex structure, such as a form in which several cells are merged or a form in which many empty cells exist. Therefore, learning data according to some embodiments of the present disclosure can increase the learning efficiency of a model for table recognition.

The steps of the method for generating training data for the table recognition model described above are presented only for explanation, and some steps may be omitted or separate steps may be added. Additionally, the steps described above may be performed in any order. Alternative steps are described below.

7 is a block diagram of a computing device according to some embodiments of the present disclosure.

7 shows a brief, general schematic diagram of an example computing environment in which some embodiments of the present disclosure may be implemented.

Although the disclosure has been described generally in the context of computer-executable instructions that can be executed on one or more computers, those skilled in the art will appreciate that the disclosure can be combined with other program modules and/or implemented as a combination of hardware and software. will be.

Typically, program modules include routines, programs, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Additionally, those skilled in the art will understand that the methods of the present disclosure are applicable to single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, hand-held computing devices, microprocessor-based or programmable consumer electronics, and the like (each of which may be It will be appreciated that the system may be implemented with other computer system configurations, including those capable of operating in conjunction with one or more associated devices.

The described embodiments of the disclosure can also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer-readable media. Any medium that can be accessed by a computer can be a computer-readable medium. Computer-readable media includes volatile and non-volatile media, transitory and non-transitory media, removable and non-removable media. By way of example, and not limitation, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media refers to volatile and non-volatile media, transient and non-transitory media, removable and non-removable, implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Includes media. Computer storage media includes RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices; or any other medium that can be accessed by a computer and used to store desired information.

Computer-readable transmission media typically implements computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as other transport mechanisms, and includes all information transmission media. do. The term modulated data signal refers to a signal in which one or more of the characteristics of the signal have been set or changed to encode information within the signal. By way of example, and not limitation, computer-readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

An example environment 1100 is shown that implements various aspects of the present disclosure, including a computer 1102, which includes a processing unit 1104, a system memory 1106, and a system bus 1108. do. System bus 1108 couples system components, including but not limited to system memory 1106, to processing unit 1104. Processing unit 1104 may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be used as processing unit 1104.

System bus 1108 may be any of several types of bus structures that may further be interconnected to a memory bus, peripheral bus, and local bus using any of a variety of commercial bus architectures. System memory 1106 includes read only memory (ROM) 1110 and random access memory (RAM) 1112. The basic input/output system (BIOS) is stored in non-volatile memory 1110, such as ROM, EPROM, and EEPROM, and is a basic input/output system that helps transfer information between components within the computer 1102, such as during startup. Contains routines. RAM 1112 may also include high-speed RAM, such as static RAM, for caching data.

Computer 1102 may also include an internal hard disk drive (HDD) 1114 (e.g., EIDE, SATA)—the internal hard disk drive 1114 may also be configured for external use within a suitable chassis (not shown). -, a magnetic floppy disk drive (FDD) 1116 (e.g., for reading from or writing to a removable diskette 1118), and an optical disk drive 1120 (e.g., a CD-ROM for reading the disk 1122 or reading from or writing to other high-capacity optical media such as DVDs). Hard disk drive 1114, magnetic disk drive 1116, and optical disk drive 1120 are connected to system bus 1108 by hard disk drive interface 1124, magnetic disk drive interface 1126, and optical drive interface 1128, respectively. ) can be connected to. The interface 1124 for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. For computer 1102, drive and media correspond to storing any data in a suitable digital format. Although the description of computer-readable media above refers to removable optical media such as HDDs, removable magnetic disks, and CDs or DVDs, those skilled in the art will also recognize removable optical media such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc. It will be appreciated that other types of computer-readable media may also be used in the exemplary operating environment, and that any such media may contain computer-executable instructions for performing the methods of the present disclosure.

A number of program modules may be stored in drives and RAM 1112, including an operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136. All or portions of the operating system, applications, modules and/or data may also be cached in RAM 1112. It will be appreciated that the present disclosure may be implemented on various commercially available operating systems or combinations of operating systems.

A user may enter commands and information into computer 1102 through one or more wired/wireless input devices, such as a keyboard 1138 and a pointing device such as mouse 1140. Other input devices (not shown) may include microphones, IR remote controls, joysticks, game pads, stylus pens, touch screens, etc. These and other input devices are connected to the processing unit 1104 through an input device interface 1142, which is often connected to the system bus 1108, but may also include a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, It can be connected by other interfaces, etc.

A monitor 1144 or other type of display device is also connected to system bus 1108 through an interface, such as a video adapter 1146. In addition to monitor 1144, computers typically include other peripheral output devices (not shown) such as speakers, printers, etc.

Computer 1102 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148, via wired and/or wireless communications. Remote computer(s) 1148 may be a workstation, computing device computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer device, or other conventional network node, and is generally connected to computer 1102. For simplicity, only memory storage device 1150 is shown, although it includes many or all of the components described. The logical connections depicted include wired/wireless connections to a local area network (LAN) 1152 and/or a larger network, such as a wide area network (WAN) 1154. These LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a worldwide computer network, such as the Internet.

When used in a LAN networking environment, computer 1102 is connected to local network 1152 through wired and/or wireless communication network interfaces or adapters 1156. Adapter 1156 may facilitate wired or wireless communication to LAN 1152, which also includes a wireless access point installed thereon for communicating with wireless adapter 1156. When used in a WAN networking environment, the computer 1102 may include a modem 1158 or be connected to a communicating computing device on the WAN 1154 or to establish communications over the WAN 1154, such as via the Internet. Have other means. Modem 1158, which may be internal or external and a wired or wireless device, is coupled to system bus 1108 via serial port interface 1142. In a networked environment, program modules described for computer 1102, or portions thereof, may be stored in remote memory/storage device 1150. It will be appreciated that the network connections shown are exemplary and that other means of establishing a communications link between computers may be used.

Computer 1102 may be associated with any wireless device or object deployed and operating in wireless communications, such as a printer, scanner, desktop and/or portable computer, portable data assistant (PDA), communications satellite, wirelessly detectable tag. Performs actions to communicate with any device or location and telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, communication may be a predefined structure as in a conventional network or may simply be ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) enables connection to the Internet, etc. without wires. Wi-Fi is a wireless technology, like cell phones, that allows these devices, such as computers, to send and receive data indoors and outdoors, anywhere within the coverage area of a base station. Wi-Fi networks use wireless technology called IEEE 802.11 (a,b,g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, the Internet, and wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz wireless bands, for example, at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band). there is.

Those skilled in the art will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols and chips that may be referenced in the above description include voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields. It can be expressed by particles or particles, or any combination thereof.

Those skilled in the art will understand that the various illustrative logical blocks, modules, processors, means, circuits and algorithm steps described in connection with the embodiments disclosed herein may be used in electronic hardware, (for convenience) It will be understood that the implementation may be implemented by various forms of program or design code (referred to herein as “software”) or a combination of both. To clearly illustrate this interoperability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been described above generally with respect to their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. A person skilled in the art of this disclosure may implement the described functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this disclosure.

The various embodiments presented herein may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term “article of manufacture” includes a computer program or media accessible from any computer-readable device. For example, computer-readable media include magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory. Includes, but is not limited to, devices (e.g., EEPROM, cards, sticks, key drives, etc.). Additionally, various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of exemplary approaches. It is to be understood that the specific order or hierarchy of steps in processes may be rearranged within the scope of the present disclosure, based on design priorities. The appended method claims present elements of the various steps in a sample order but are not meant to be limited to the particular order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the disclosure. Thus, the present disclosure is not limited to the embodiments presented herein but is to be interpreted in the broadest scope consistent with the principles and novel features presented herein.

Claims (14)

In a method of generating training data for a model for table recognition performed by a computing device,
generating a table including a plurality of zones;
establishing a first header zone and a first content zone on the plurality of zones;
generating at least one of a horizontal header, a vertical header, and a merge header in the first header area;
merging or splitting cells on the first content area, or establishing a second header area and a second content area; and
determining a table structure based at least on the first header section and the first content section;
Including,
method.
According to claim 1,
The step of creating at least one of a horizontal header, a vertical header, and a merge header in the first header area:
generating a first row header including at least one column section on a first row among R rows for generating a horizontal header among the first header sections; and
On the nth row among the R rows, an nth row header is generated by dividing each zone corresponding to at least one zone included in the n-1th row header of the preceding n-1th row into one or more zones. steps;
Including,
method.
According to claim 2,
The step of creating at least one of a horizontal header, a vertical header, and a merge header in the first header area:
merging at least a portion of adjacent regions with the same row width on at least two consecutive rows among the R rows;
Including,
method.
According to claim 1,
The step of creating at least one of a horizontal header, a vertical header, and a merge header in the first header area:
generating a first column header including at least one row section on a first column of the C columns for generating a vertical header among the first header sections; and
On the m-th column of the C columns, generating the m-th row header by dividing each zone corresponding to at least one zone included in the m-1-th column header of the preceding m-1-th column into one or more zones. step;
Including,
method.
According to claim 4,
The step of creating at least one of a horizontal header, a vertical header, and a merge header in the first header area:
merging at least a portion of adjacent regions of the same column width in at least two consecutive columns among the C columns;
Containing more,
method.
According to claim 1,
The step of creating at least one of a horizontal header, a vertical header, and a merge header in the first header area:
Generating a merged header section by merging overlapping sections of rows 1 to R and columns 1 to C of the first header section;
Including,
method.
According to claim 1,
Merging or splitting cells on the first content area, or setting a second header area and a second content area:
dividing the first content area to correspond to the generated horizontal header, vertical header, or merge header;
Including,
method.
According to claim 7,
Merging or splitting cells on the first content area, or setting a second header area and a second content area:
merging at least a portion of the divided first content areas;
dividing at least a portion of the divided first content zones into at least one zone; or
setting a second header area and a second content area in at least some of the divided first content areas;
Containing more,
method.
According to claim 1,
setting pixel values of rows and columns of the determined table structure;
Containing more,
method.
According to claim 1,
setting text on at least some of the cells included in the determined table structure;
Containing more,
method.
According to claim 1,
setting a style on at least some of the cells included in the determined table structure;
Containing more,
method.
According to claim 11,
The style includes at least one of a border style, a text font, or a background color,
method.
A computer program stored on a computer-readable medium, the computer program comprising instructions for causing one or more processors to perform a method of generating training data for a model for table recognition, the method comprising:
generating a table including a plurality of zones;
establishing a first header zone and a first content zone on the plurality of zones;
generating at least one of a horizontal header, a vertical header, and a merge header in the first header area;
merging or splitting cells on the first content area, or establishing a second header area and a second content area; and
determining a table structure based at least on the first header section and the first content section;
Including,
A computer program stored on a computer-readable medium.
A computing device that performs a method of generating training data for a model for table recognition, the computing device comprising:
memory containing computer-executable components;
a processor executing the following computer-executable components stored in memory; Including,
The processor,
Create a table containing multiple zones,
Establishing a first header zone and a first content zone on the plurality of zones,
Generating at least one of a horizontal header, a vertical header, and a merge header in the first header area,
Merge or split cells on the first content area, or set a second header area and a second content area, and
determining a table structure based at least on the first header section and the first content section,
Computing device.