KR102486105B1 - Method for managing training data for optical character recognition - Google Patents

Abstract

According to one embodiment of the present disclosure, a computer program stored in a computer readable storage medium is disclosed. The computer program, when executed in one or more processors, performs the following operations for generating a document image using at least one network function, the operations comprising: inputting at least one or more digital document images to a first generation network; obtaining an actualized digital document image; acquiring a restored digital document image by inputting the actualized digital document image to a second generation network; learning the first identification network to identify the two document images by inputting the actualized digital document image and the actual document image into a first identification network; learning a first generation network and a second generation network based at least in part on a comparison result between the digital document image and the restored digital document image and an identification result of the first identification network, and using the learned network An operation of generating a document image may be included.

Description

Training data management method for optical character recognition {METHOD FOR MANAGING TRAINING DATA FOR OPTICAL CHARACTER RECOGNITION}

The present disclosure relates to machine learning, and more specifically, to a learning data management method for machine learning.

Artificial neural networks need to learn about data, and the performance of artificial neural networks is determined by the quality and quantity of learning data. Similarly, in the field of OCR using artificial neural networks, AI-based character recognition technology is highly dependent on the quantity and quality of learning data. Therefore, the more learning data can be secured, the better the performance of AI-based character recognition technology can be.

Korean Registered Patent Publication KR10-2003221 discloses “a writing image data generating system and a writing image data generating method”.

The present disclosure has been made in response to the above background art, and is to provide a learning data management method for machine learning.

According to one embodiment of the present disclosure for realizing the above object, a computer program stored in a computer readable storage medium is disclosed. The computer program, when executed in one or more processors, performs the following operations for generating a document image using at least one network function, the operations comprising: inputting at least one or more digital document images to a first generation network; obtaining an actualized digital document image; acquiring a restored digital document image by inputting the actualized digital document image to a second generation network; learning the first identification network to identify the two document images by inputting the actualized digital document image and the actual document image into a first identification network; learning a first generation network and a second generation network based at least in part on a comparison result of the digital document image and the restored digital document image and an identification result of the first identification network; and generating a document image using the learned generation network.

In an alternative embodiment of computer program operations for performing the following operations for generating a document image, the at least one digital document image is based on digital document image generating material including elements for generating a digital document image. can be created

In an alternative embodiment of the computer program operations performing the following operations for generating a document image, the at least one digital document image may be obtained by crawling published digital document material.

In an alternative embodiment of computer program operations for performing the following operations for generating a document image, the actualized digital document image is an image for which style conversion has been performed on the digital document image by the first generation network. , has a style similar to that of the actual document image, and the restored digital document image is an image for which style conversion has been performed on the actualized digital document image by the second generation network, and has a style similar to that of the digital document image. can have

In an alternative embodiment of the computer program operations for performing the following operations for generating a document image, the second generating network is inversely related to the first generating network and has a network structure that is a mirror image of the first generating network. can have

In an alternative embodiment of the computer program operations for performing the following operations for generating a document image, the first identification network is configured for each of the actualized digital document image and the real document image to identify the two inputted document images. It may include an operation for giving a probability value indicating whether or not it is an actual document image.

In an alternative embodiment of computer program operations for performing the following operations for generating a document image, the first identification network may include a probability value indicating whether the actualized digital document image is a real document image and a real document image for the actual document image. It can be learned using a loss function based on a probability value representing whether or not a document image exists.

In an alternative embodiment of the computer program operations for performing the following operations for generating a document image, the first generation network causes the first identification network to transmit the actualized digital document image generated in the first generation network and the actual It can be taught not to identify document images.

In an alternative embodiment of the computer program operations for performing the following operations for generating a document image, the generating of the document image inputs the digital document image to the first trained network to generate an actualized digital document image It may include an operation to generate.

In an alternative embodiment of the computer program operations for performing the following operations for generating a document image, the operation of generating the document image inputs a real document image to the learned second generation network to obtain a restored real document image. It may include an operation to generate.

In an alternative embodiment of computer program operations for performing the following operations for generating a document image, inputting at least one real document image to a second generation network to obtain a restored real document image, the restored real document image acquiring a real document image by inputting a document image into the first generation network, and inputting the restored real document image and the digital document image into a second identification network to identify the two document images; and learning a second generation network based at least in part on a comparison result between the real document image and the actualized real document image and an identification result of the second identification network.

In an alternative embodiment of computer program operations for performing the following operations for generating a document image, the at least one real document image may be obtained by crawling real document images published on the Internet.

In an alternative embodiment of computer program operations for performing the following operations for generating a document image, the restored real document image is an image for which style conversion is performed on the real document image by the second generation network, and The actual document image having a style similar to that of the digital document image may have a style similar to the actual document image as an image for which style conversion is performed on the restored real document image by the first generation network.

In an alternative embodiment of the computer program operations for performing the following operations for generating a document image, the second identification network is configured for each of the restored real document image and digital document image to identify the two input document images. It may include an operation for giving a probability value indicating whether or not it is a digital document image.

In an alternative embodiment of the computer program operations for performing the following operations for generating a document image, the second identification network may include a probability value indicating whether the restored real document image is a digital document image and a digital document image for the digital document image. It can be learned using a loss function based on a probability value representing whether or not a document image exists.

In an alternative embodiment of computer program operations performing the following operations for generating a document image, the second generating network may be trained such that the second identification network does not discriminate between a reconstructed real document image and a digital document image. can

A method for generating a document image is disclosed according to an embodiment of the present disclosure for realizing the above object. A method for generating a document image includes: inputting at least one digital document image into a first generating network to obtain a realized digital document image; acquiring a restored digital document image by inputting the actualized digital document image to a second generation network; learning the first identification network to identify the two document images by inputting the actualized digital document image and the real document image into a first identification network; learning a first generation network and a second generation network based at least in part on a comparison result of the digital document image and the restored digital document image and an identification result of the first identification network; and acquiring a realized digital document image by inputting the digital document image to the first generated network.

A document image generator is disclosed according to an embodiment of the present disclosure for realizing the above object. The generator may include one or more processors for processing document image data; and a memory for storing at least one deep learning-based network function, wherein the one or more processors obtain a realized digital document image by inputting at least one or more digital document images to a first generation network, A digital document image is input to a second generation network to obtain a restored digital document image, and the actualized digital document image and the actual document image are input to a first identification network to learn the first identification network to identify the two document images. learning a first generation network based at least in part on a comparison result of the digital document image with the restored digital document image and an identification result of the first identification network; A realized digital document image can be acquired by inputting it into the network.

According to one embodiment of the present disclosure for realizing the above object, a computer readable recording medium having a data structure for storing data related to the operation of an artificial neural network is stored. The data is generated through the following operations, which include: inputting at least one or more digital document images to the first generation network to obtain a realized digital document image; acquiring a restored digital document image by inputting the actualized digital document image to a second generation network; learning the first identification network to identify the two document images by inputting the actualized digital document image and the actual document image into a first identification network; learning a first generation network and a second generation network based at least in part on a comparison result of the digital document image and the restored digital document image and an identification result of the first identification network; and acquiring a realized digital document image by inputting the digital document image to the first generated network.

The present disclosure may provide a learning data management method for machine learning.

1 is a block diagram of a computing device for generating a document image using at least one network function according to an embodiment of the present disclosure.
2 is a diagram illustrating examples of types of document images according to an embodiment of the present disclosure.
3 is a conceptual diagram illustrating a learning process between a first generation network and a first identification network and a relationship between networks according to an embodiment of the present disclosure.
4 is a conceptual diagram illustrating a relationship between networks in a learning data generator used for AI-based character recognition including a first generation network, a second generation network, and a first identification network according to an embodiment of the present disclosure.
5 is a conceptual diagram illustrating a relationship between networks when a second identification network is used to learn a second generation network according to an embodiment of the present disclosure.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the present disclosure. However, it is apparent that these embodiments may be practiced without these specific details.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless otherwise specified or clear from the context, “X employs A or B” is intended to mean one of the natural inclusive substitutions. That is, X uses A; X uses B; Or, if X uses both A and B, "X uses either A or B" may apply to either of these cases. Also, the term "and/or" as used herein should be understood to refer to and include all possible combinations of one or more of the listed related items.

Also, the terms "comprises" and/or "comprising" should be understood to mean that the features and/or components are present. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, elements, and/or groups thereof. Also, unless otherwise specified or where the context clearly indicates that a singular form is indicated, the singular in this specification and claims should generally be construed to mean "one or more".

In addition, the term “at least one of A or B” should be interpreted as meaning “when only A is included”, “when only B is included”, and “when A and B are combined”.

Those skilled in the art will further understand that the various illustrative logical blocks, components, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented using electronic hardware, computer software, or combinations of both. It should be recognized that it can be implemented as To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or as software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of this disclosure.

The description of the presented embodiments is provided to enable any person skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art of this disclosure. The general principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. Thus, the present invention is not limited to the embodiments presented herein. The present invention is to be accorded the widest scope consistent with the principles and novel features set forth herein.

Throughout this specification, a digital document image includes a document created based on elements for creating a document in a computing device. The digital document image includes a document image captured and/or stored as a screen on which a character string generated through a text editor or the like is output to the computing device. That is, in this specification, a digital document image refers to a document image completely generated by an electrical signal inside a computing device.

In this specification, the actual document image includes a document image obtained by photographing a document created through a method such as outputting a document created outside a computing device or generated by a document editor of a computing device. The actual document image includes a photographed image of various documents that a user may encounter in real life. In addition, the actual document image includes, for example, an image of a document in which a printed character string and a character string handwritten by a user are mixed. In addition, the actual document image may contain noise regarding character strings. Furthermore, the actual document image may not have a constant illuminance distribution within the entire document image.

The actualized document image in this specification includes a document image created according to an embodiment of the present disclosure. In the present disclosure, a materialized document image is a concept including a materialized digital document image and a materialized real document image. The actualized digital document image may be an image obtained as an output by inputting a digital document image to a first generation network to be described later. The actualized real document image may be an image obtained as an output of inputting the restored real document image to the first generating network. Throughout this specification, the word “actualized” may be used to mean “obtained as an output of a first generating network”, “generated from a first generating network”, and the like. In this specification, a realized document image means a document image having a style similar to that of an actual document image. The style includes image resolution, image noise level, color temperature included in the image, type of color included in the image, type of line, thickness of the line, color tone, and the like. The foregoing style of description is exemplary only and the present disclosure is not limited thereto.

In this specification, a restored document image is a concept including a restored digital document image and a restored actual document image. The restored digital document image may be an image obtained as an output by inputting the actualized digital document image to a second generation network to be described later. The restored real document image may be an image obtained as an output of inputting the real document image to the second generating network. Throughout this specification, the word “reconstructed” may be used in the same sense as “obtained as an output of the second generation network” or “generated from the second generation network”. In this specification, a restored document image means a document image having a style similar to that of a digital document image.

1 is a block diagram of a computing device for generating a document image using at least one network function according to an embodiment of the present disclosure.

The configuration of the computing device 100 shown in FIG. 1 is only a simplified example. In one embodiment of the present disclosure, the computing device 100 may include other components for performing a computing environment of the computing device 100, and only some of the disclosed components may constitute the computing device 100. The computing device 100 may include a processor 110 , a memory 130 , and a network unit 150 .

The processor 110 may input one or more digital document images to the first generation network to obtain a realized digital document image. The learning method of the first generation network will be described in detail later.

The at least one digital document image may be created based on digital document image generating data including elements for generating a digital document image. Specifically, the digital image generation data may include properties such as character string, text color, text size, text font, type of line, and background style. For example, if there are N strings stored in memory (string 1, string 2, ... string N), M fonts stored in memory, and K background styles of digital documents stored in memory, Processor 110 may generate up to N*M*K different digital document images. The processor 110 may generate different digital document images based on digital image generating data based on a random function. The random function may be a function that randomly selects data from a given data set using a specific seed value or a current time as a seed. The specific description of the aforementioned random function is only an example, and the present disclosure is not limited thereto.

The processor 110 may acquire at least one digital document image by crawling open digital document data. The crawling technique detects an image tag on a code of a web page to identify a document image, and stores a related document image to obtain an actual document image. The crawling technique may obtain a digital document image by directly outputting a screen displayed on a webpage screen regardless of an inserted image. The specific description of the aforementioned crawling technique is only an example, and the present disclosure is not limited thereto. The processor 110 obtains digital document images and actual document images released through the crawling method more quickly and in various ways and uses them as training data for learning the generation network and the identification network, thereby generating the generation network and the identification network through the rich learning data. performance can be further improved. In one embodiment of the present disclosure, for example, the digital document image may include images of documents related to the financial industry, but the present disclosure is not limited thereto.

In the training of artificial neural networks, the quantity and quality of input data is critical to the performance of the trained artificial neural networks. In the present invention, the processor 110 generates various digital document images based on digital document image generation data or acquires digital document images by crawling open digital document data, so that digital document images in various fields are quickly obtained and artificial neural networks are acquired. provides a lot of input data.

The processor 110 may obtain a restored digital document image by inputting the actualized digital document image to the second generating network. The actualized digital document image refers to a style-converted document image obtained as an output by inputting a digital document image to a first generation network to have a style similar to that of the actual document image. The restored digital document image refers to a style-converted document image that is input to a second generation network to have a style similar to that of the digital document image again, and obtained as an output as a result.

The second generation network may be a generation network having an inverse function relationship with the first generation network. The structure of the second generation network may have a mirror image structure of the first generation network in order to act as an inverse function of the first generation network. The second generation network may have a structure in which input and output are inverted from those of the first generation network. Specifically, the second generation network may be a network function learned to take an output of the first generation network as an input and output data similar to the input of the first generation network. That is, the first generating network receives a digital document image as an input and outputs a realized digital document image similar to the real document image, and the second generating network receives a real document image as an input and outputs a restored real document image similar to the digital document image. can output As an embodiment in which the second generating network has a mirror image structure of the first generating network, when the first generating network is a network function of an autoencoder structure composed of encoders and decoders, the second generating network is a mirror image of the first generating network. It may be a network function having a decoder-encoder structure as a structure. In this case, each of the decoder-encoders of the second generation network is one in which the input layer and the output layer are sequentially inverted (i.e., the output layer of the decoder is the input layer of the second generation network, and the input stage of the decoder is the second It may be the bottleneck layer of the generation network, the input layer of the encoder may be the output layer of the second generation network, and the output layer of the encoder may be the bottleneck layer of the second generation network. The learning method of the second generation network will be described in detail below.

The processor 110 may learn the first identification network to identify the two document images by inputting the actualized digital document image and the real document image to the first identification network. At this time, learning the first identification network to identify the two document images includes learning differences to distinguish the two document images. Hereinafter, a method for the processor 110 to learn the first identification network to generate learning data used for AI-based character recognition will be described.

Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network includes one or more nodes. Nodes (or neurons) constituting neural networks may be interconnected by one or more links.

The first generative network may be learned adversarially by using a generative adversarial network (GAN) by configuring the first identification network and the adversarial network. The first generation network may include at least one artificial neural network and learn to have a style similar to an actual document for an input document image. The first identification network compares the document image style-converted by the first generation network, ie, the actualized document image, with the actual document image to determine whether they are similar.

The first generation network learned in a mutually adversarial manner by the processor 110 may utilize the identification result of the first identification network as its objective function to calculate an error for back propagation. The first identification network may include an operation for assigning a probability value indicating whether or not the actual document image is a real document image to each of the actualized digital document image and the actual document image. Specifically, the first identification network sets a predetermined interval for the probability value indicating whether or not the actual document image is present, and when the style of the actual document image and the input document image are similar, a high value is given according to the degree of similarity in the predetermined interval. and when the style of the actual document image and the input document image are different, it may be learned in a method of assigning a low value according to the degree of similarity in a predetermined section. Here, the first identification network can be learned in a way that gives an opposite probability value. In other words, when the style is similar to the actual document image, a low value is given according to the degree of similarity in a predetermined section, and when the style of the actual document image and the input document image are different, a high value is given according to the degree of similarity in a predetermined section. It can also be learned in a way.

In one embodiment according to the present disclosure, the processor 110 determines whether the first identification network is a real document image for a realized digital document image and a probability value indicating whether a real document image is a real document image. It can be learned using a loss function based on The actualized digital document image may be an image output by the first generating network. The loss function is a function used to optimize the entire artificial neural network. In this specification, the loss function and the objective function may be used in the same meaning.

The processor 110 may use the following function as a loss function based on a probability value for each document image indicating whether or not the document image is a real document image.

(loss function 1)

In the loss function 1, D is a function representing the first identification network, and G is a function representing the first generating network. y represents the actual document image input to the first identification network. x represents the digital document image input to the first generation network, so G(x) represents the actualized digital document image that is the output of the first generation network. The processor 110 assigns a probability value indicating whether or not the document image is a real document image to each document image by inputting y and G(x) to the first identification network.

의 결과값이 최대가 되도록 제 1 식별 네트워크(D)를 학습한다. 예를 들어, 학습을 통해 식별 네트워크가 실제 문서 이미지와 실제화된 디지털 문서 이미지를 잘 구별할 수 있게 되면, 실제 문서 이미지인 입력 y에 대해서는 1에 가까운 확률값(D(y))을 부여하게 될 것이다. 또한 실제화된 디지털 문서 이미지인 입력 G(x)에 대해서는 0에 가까운 확률값(D(G(x))을 부여하게 될 것이다. 그 결과 상기 손실함수 1에 포함된 두 log 함수는 두 함수 모두 입력으로 1에 가까운 값을 얻게 되어 0에 가까운 함수값을 출력하게 된다. 결과적으로, 입력이 [0,1] 구간으로 제한된 log 함수의 결과값의 최대값은 0이므로, 상기 손실함수 1의 최대값은 0이 된다. 그러므로 프로세서(110)는 손실함수 1을 이용하여 제 1 식별 네트워크가 손실함수의 결과값이 최대가 되도록 학습할 수 있다. 여기서 손실함수 1은 일 예시에 불과하여 본원 발명은 이에 제한되지 않으며, 실제 문서 이미지와 실제화된 디지털 문서 이미지와 같이 구별하고자 하는 두 문서 이미지에 대해 사전 결정된 구간 내에서 구간의 양 끝 값을 확률값으로 부여하는 손실함수를 생성한 뒤, 프로세서(110)는 제 1 식별 네트워크가 그 손실함수의 결과값이 최대가 되도록 학습할 수 있다.The processor 110 may train the first identification network to distinguish between real document images and actualized digital document images. Specifically, the processor 110 uses the first identification network to assign a high probability value indicating whether or not a real document image is input to a real document image input to the first identification network, and to assign a low probability value to an input actualized digital document image. can learn The processor 110 may learn the first identification network to maximize the resulting value of the loss function. In other words,

The first identification network (D) is learned so that the result value of is maximized. For example, if the identification network can distinguish a real document image from a real digital document image through training, it will give a probability value (D(y)) close to 1 for the input y, which is a real document image. . In addition, a probability value D(G(x)) close to 0 will be given to the input G(x), which is a realized digital document image. As a result, the two log functions included in the loss function 1 are both inputs. A value close to 1 is obtained and a function value close to 0 is outputted. 0. Therefore, the processor 110 can learn the first identification network to maximize the resultant value of the loss function by using the loss function 1. Here, the loss function 1 is only an example and the present invention is limited thereto. After generating a loss function that assigns values at both ends of the interval as probability values within a predetermined interval for two document images to be distinguished, such as a real document image and a real digital document image, the processor 110 1 The identification network can learn to maximize the output of its loss function.

Processor 110 may learn a first generation network and a second generation network based at least in part on a comparison result of the digital document image and the reconstructed digital document image and an identification result of the first identification network. Here, the identification result of the first identification network may include a probability value indicating whether or not the actual document image is a real document image that the first identification network assigns to each of the actualized digital document image and the real document image.

The following describes a method for learning a first generating network based at least in part on an identification result of the first identifying network.

The processor 110 may mutually learn the first identification network and the first generation network based on a GAN scheme. The processor 110 may learn the first generation network so that the first identification network does not discriminate between real document images and actualized digital document images. Specifically, the first generation network may learn to derive an opposite result for the same loss function as the first identification network. For example, if the first identification network is trained to have a maximum loss function, the first generation network can be trained to have a minimum loss function. Conversely, when the first identification network is trained to have a minimum loss function, the first generation network can be trained to have a maximum loss function.

의 결과값이 최소가 되도록 제 1 생성 네트워크를 학습할 수 있다. 구체적으로, 손실함수 1의 결과값이 최소가 되도록 제 1 생성 네트워크를 학습한다는 것은, 실제화된 디지털 문서 이미지(G(x))에 대해 제 1 식별 네트워크(D)가 실제 문서 이미지와 스타일 유사도가 높다고 판단해 높은 확률값을 부여하는 경우를 말한다. 프로세서(110)가 제 1 생성 네트워크를 상기한 바와 같이 학습하는 경우

부분은

에 가까이 수렴하게 되므로 그 값은 음의 무한대로 발산하게 되어 결과적으로 손실함수는 최소값을 가지게 된다. 전술한 손실함수의 예시는 일 예시에 불과하며 본 개시는 이에 제한되지 않는다.The processor 110 may learn the first generation network so that the first identification network that derives the identification result based on the above loss function 1 does not discriminate between the real document image and the actualized digital document image. When the first identification network is learned to derive the maximum value of the loss function 1, the processor 110 may learn the first generation network to have the minimum value of the loss function 1. In other words,

The first generating network can be learned so that the output value of is minimized. Specifically, learning the first generation network so that the result value of the loss function 1 is minimized means that the first identification network (D) has a style similarity with the actual document image for the actualized digital document image (G(x)). This refers to the case in which a high probability value is assigned because it is judged to be high. When the processor 110 learns the first generation network as described above

part is

Since it converges close to , the value diverges to negative infinity, and as a result, the loss function has a minimum value. The above-described example of the loss function is only an example, and the present disclosure is not limited thereto.

As described above, by mutually learning the first generation network and the first identification network in the GAN method, the present invention for generating a document image can generate the first generation network in an unsupervised learning method without labeling true values. there is.

Hereinafter, a method for learning a first generation network and a second generation network based at least in part on a comparison result between the digital document image and the restored digital document image will be described.

According to an embodiment of the present disclosure, the processor 110 determines a first generation network and a second generation network based on a comparison result between the digital document image and the restored digital document image, in addition to the identification result of the first identification network. can learn The processor 110 may create an objective function required to learn the first generation network and the second generation network based on the comparison result. Specifically, a comparison result between the digital document image and the restored digital document image may mean a difference in RGB values between pixels. Also, the comparison result may refer to a result of a loss function having a digital document image as a true value and a restored digital document image generated by the first generation network and the second generation network as final predicted values. Here, the loss function may mean the mean squared error (MSE) between the true value and the predicted value. Also, cross entropy can be used as a loss function. Furthermore, the comparison result may be derived by learning the difference between the digital document image and the restored digital document image through a separate network function.

According to an embodiment of the present disclosure, a first generation network and a second generation network are learned based on a comparison result between the digital document image and the restored digital document image, so that the second generation network is configured to obtain at least the first generation network. The style of the output document image is maintained and learned so that it can be converted or restored to the style of the input document image.

In general, a network function derives output data by predicting input data, and compares the predicted value with the true value to modify the output data. Specifically, for example, the digital document image input to the first generating network may be a document image including the character string “A”. At this time, the first generating network may output the actualized document image for “A” as an output. After that, when a realized document image for “A” and an actual document image for “B” are input to the first identification network to identify the two document images, the first generation network identifies two document images. It can be learned to change “a” to “b”, that is, to change a character string so that document images cannot be distinguished.

However, according to the present disclosure, since the first generation network maintains a level at which the output data of the first generation network can be restored to the input data style by the second generation network and learns the style change, the input data It is learned to change the style while avoiding changes to the extent of losing its unique characteristics. Therefore, according to the present disclosure, the first generating network can actualize a digital document image and learn the first generating network even if there is no correct image corresponding to the actualized digital document image. This has the advantage of being able to learn the AI-based character recognition learning data generator even when the amount of training data is insufficient, including when there is no correct image of the document image after style conversion for the document image before style conversion.

The processor 110 may train the first generation network and the second generation network at the same time by backpropagating an error according to a comparison result between the digital document image and the restored digital document image. In other words, when the first generation network and the second generation network are viewed as one network function, the one network function is a network function that receives a digital document image as an input and outputs a restored digital document image. Accordingly, the one network function may be learned based on an error between an input digital document image and an output digital document image. The process of learning one network function may simultaneously learn the first generation network and the second generation network.

The processor 110 may train the second generation network based on a comparison result between the digital document image and the restored digital document image. Specifically, the processor 110 fixes weights, bias values, etc. included in the model of the first generation network, and outputs the digital document image as input data of the first generation network and the restored digital document as output data of the second generation network. A second generation network may be trained based on the comparison result of the images.

The processor 110 may train a second generation network based on the identification result of the first identification network. Specifically, the processor 110 may obtain a restored real document image by inputting the real document image to the second generation network, and then input the restored real document image to the first generation network to obtain a real document image. In addition, a second generation network may be learned by inputting the actualized real document image and the actual document image into the learned first identification network and back-propagating the error to the second generation network.

The present disclosure provides a second generation network that changes the style of the output of the first generation network back to the style of the input of the first generation network, so that the first generation network styles a digital document image, which is an initial input, into an actual document image. There is an effect of preventing excessive loss of information in the process of conversion and preventing overfitting to an actual document image style.

The processor 110 may generate a document image using the learned generation network. The learned generation network includes a first generation network and a second generation network. The operation of generating the document image by the processor 110 may be an operation of inputting the digital document image to the first learned generation network to generate a realized digital document image. The actualized digital document image obtained in this way can be rendered indistinguishable from the actual document image by an artificial neural network and provided as learning data to an AI-based text recognition technology. As another embodiment, the operation of generating the document image by the processor 110 may be an operation of generating a restored real document image by inputting the actual document image to the learned second generation network. As described above, the document image reconstructed by the second generation network is a document image having a style similar to that of the digital document image, and noise existing in the actual document image can be removed through restoration of the actual document image. In addition, style standardization of actual document images collected through the second generating network is possible. Here, style standardization includes standardization into a digital document format by restoring an actual document image to which different styles are applied to a digital document image form. Furthermore, when an actual document image is input to the second generation network and the output is used for an OCR technology including character position recognition and character recognition technology, a data purification effect including noise removal on the actual document image can be obtained. Therefore, the performance of OCR technology can be maximized. In other words, it has the advantage of functioning as a preprocessor for effective character recognition.

2 is a diagram illustrating examples of types of document images according to an embodiment of the present disclosure.

When the computing device inputs the digital document image 201 generated for the character string “overlapping type” into the first generation network, it can obtain a digital document image 202 actualized for the string “overlapping type” as an output thereof. there is. Alternatively, the computing device inputs the real document image 204 for the string “/h.h/o” into the second generation network and outputs the restored real document image 203 for the string “/h.h/o”. can be obtained In this way, when the digital document image is input to the first generation network, a realized digital document image having a style similar to the actual document image can be generated, and when the actual document image is input to the second generation network, the digital document image It is possible to create a restored real document image with a style similar to The above-described example of the character string is only an example, and the present disclosure is not limited thereto.

3 is a conceptual diagram illustrating a learning process between a first generation network and a first identification network and a relationship between networks according to an embodiment of the present disclosure. The solid line 400 in FIG. 3 represents the flow of data input and output to each network, and the broken line 401 represents that the generation network is learned based on the identification results of the two document images of the identification network. When at least one digital document image 301 is input to the first generating network 311, a realized digital document image 303 is output. The actualized digital document image 303 may then be input to the first identification network 312 together with the actual document image 302 . The first identification network may assign a probability value indicating whether or not each input document image is a real document image, and may be trained to better discriminate it. As described above, the fact that the first identification network is trained to discriminate between two document images may include an operation for assigning a high probability value to an actual document image and a low probability value to an actualized document image. Based on the results of the first identification network 312, the first generation network 311 can be trained so that the first identification network does not distinguish between the two images well. That is, the first generation network may be trained to assign a high probability value indicating whether the first identification network is a real document image to the actualized digital document image.

4 is a conceptual diagram illustrating a relationship between networks in a learning data generator used for AI-based character recognition including a first generation network, a second generation network, and a first identification network according to an embodiment of the present disclosure. In FIG. 4 , a solid line 400 represents a flow of data input and output to each network, and a broken line 401 in FIG. 4 indicates that the generation network is learned based on the identification results of two document images of the identification network. Further, a dashed-dotted line 402 indicates that a final output document image is compared with an initial input document image to derive a comparison result when the first generation network and the second generation network are regarded as one network function. Based on the comparison result, the first generation network and the second generation network may be learned. The first generation network 311 determines the comparison result between the restored digital document image 305 output from the second generation network 313 and the initially input digital document image 301 and the identification result of the first identification network 312. can be learned based, at least in part. The first identification network 312 may receive the actualized digital document image 303 and the actual document image 302 output from the first generation network 311 and identify the two document images. The second generation network 313 may receive the actualized digital document image 303 that is an output of the first generation network 311 and derive a restored digital document image 305 as an output thereof. The second generation network 313 may be learned based on a comparison result between the restored digital document image 305 output from the second generation network 313 and the digital document image 301 input to the first generation network. . The second generating network 313 may be learned simultaneously with the first generating network 311 , or the first generating network 311 may be learned alone while being fixed.

Hereinafter, a method of using the second identification network for effective learning of the second generation network is described.

According to an embodiment of the present disclosure, the processor 110 may use the second identification network to learn the second generation network. Specifically, as described above, the processor 110 may symmetrically learn the second generation network and the second identification network in response to mutual learning of the first generation network and the first identification network. The processor 110 may obtain a restored real document image by inputting at least one real document image to the second generation network. The processor 110 may acquire a real document image by inputting the restored real document image to the first generation network. The processor 110 may input the restored real document image and the digital document image to a second identification network to train the second identification network to identify the two document images. Furthermore, a second generation network may be learned based at least in part on a comparison result between the actual document image and the actualized actual document image and an identification result of the second identification network.

The processor 110 may learn the second generation network and the second identification network using the aforementioned GAN method. The second identification network by the processor 110 includes an operation for assigning a probability value indicating whether or not the document image is a digital document image to each of the restored real document image and the digital document image in order to identify the two input document images. That is, the second identification network identifies the two document images by assigning a probability value indicating whether or not they are digital document images to each of the digital document image and the restored real document image, which is an output of the second generation network. In addition, the second identification network by the processor 110 learns using a loss function based on a probability value indicating whether or not a restored real document image is a digital document image and a probability value indicating whether or not a digital document image is a digital document image. It can be. This corresponds to the learning method of the first identification network described above.

Hereinafter, a method for learning the second generation network and the second identification network will be described with reference to FIG. 5 in detail. 5 is a conceptual diagram illustrating a relationship between networks when a second identification network is used to learn a second generation network according to an embodiment of the present disclosure. In FIG. 5, a solid line 400 indicates the type of data input and output to each network, and a broken line 401 in FIG. 5 indicates that the generation network is learned based on the identification results of two document images of the identification network. Further, a dashed-dotted line 402 indicates that a final output document image is compared with an initial input document image to derive a comparison result when the first generation network and the second generation network are regarded as one network function. Based on the comparison result, the first generation network and the second generation network may be learned.

In FIG. 5 , the processor 110 may input one or more real document images 302 to the second generating network 313 and output a restored real document image 304 . Here, the restored actual document image 304 means an image having a style similar to the digital document image 301 of the present specification. The processor 110 inputs the restored real document image 304 and digital document image 301 to the second identification network 314 to learn the second identification network 314 to identify the two document images. can The identification result of the second identification network is reflected in the second generation network 313, and the second generation network 313 prevents the second identification network 314 from distinguishing two document images, that is, the second identification network 313 may be learned to determine a restored real document image as a digital document image and give a high probability value indicating whether or not it is a digital document image. The processor 110 may obtain a real document image 306 by inputting the restored real document image 304 output from the second generating network to the first generating network 311 . The actualized real document image is a document image in which a real document image is reconstructed similarly to a digital document image and then realized again. This is the resulting document image. The second generating network may be learned based on a comparison result between the actualized real document image and the original input real document image in addition to the identification result of the second identification network. The specific method may correspond to learning of the above-described first generation network based on a comparison result between the digital document image and the restored digital document image.

As described above, when the second generation network is learned to correspond to the first generation network through the second identification network in a manner corresponding to the first generation network, the performance of the second generation network is improved compared to the case of using only the first identification network. . Additionally, the second identification network augments the performance of the second generation network to perform style conversion from real document images to digital document images. Further, in addition to learning the combination of the first and second generating networks that take a digital document image as an input and a restored digital document image as an output, a real document image as an input and a real document image as an output can be learned. Since it is possible to learn a combination of the second and first generation networks, the performance of each generation network can be improved.

In the present disclosure, the first generation network is not only mutually learned by the first identification network in a GAN manner, but also learned while considering the constraint that it must be possible to convert into the original digital document image style by the second generation network of the mirror image. It has the advantage of being able to convert the style into an actual document image without losing the main properties of the first input digital document image. Similarly, the second generating network can be learned based on the first identifying network together with the first generating network or based on an additional second identifying network. The second generation network is a network that style-converts an actual document image or a realized document image into a digital document image, and converts a digital document image to be provided to the character recognition technology through style conversion including noise removal from the actual document image. can create

Document images generated according to the present disclosure may be divided into document images generated by the first generation network and document images generated by the second generation network. At this time, the document image generated by the first generation network can be used as learning data used for AI-based character recognition, and since it is indistinguishable from real documents by a computing device, it is required to prepare a large amount of real documents. It has the effect of saving time and money. That is, the first generation network may function as a learning data generator to be provided to an AI-based text recognition technology.

On the other hand, the document image generated by the second generation network may be a document image of relatively clear picture quality and quality because a real document image is style-converted into a digital document image. Such a second generating network may function as a preprocessor for providing high-quality document images to text recognition technology.

According to an embodiment of the present disclosure, a computer readable medium storing a data structure is disclosed.

Data structure can refer to the organization, management, and storage of data that enables efficient access and modification of data. Data structure may refer to the organization of data to solve a specific problem (eg, data retrieval, data storage, data modification in the shortest time). A data structure may be defined as a physical or logical relationship between data elements designed to support a specific data processing function. A logical relationship between data elements may include a connection relationship between user-defined data elements. A physical relationship between data elements may include an actual relationship between data elements physically stored in a computer-readable storage medium (eg, a persistent storage device). The data structure may specifically include a set of data, a relationship between data, and a function or command applicable to the data. Through an effectively designed data structure, a computing device can perform calculations while using minimal resources of the computing device. Specifically, the computing device can increase the efficiency of operation, reading, insertion, deletion, comparison, exchange, and search through an effectively designed data structure.

The data structure can be divided into a linear data structure and a non-linear data structure according to the shape of the data structure. A linear data structure may be a structure in which only one data is connected after one data. Linear data structures may include lists, stacks, queues, and decks. A list may refer to a series of data sets in which order exists internally. The list may include a linked list. A linked list may be a data structure in which data are connected in such a way that each data is connected in a single line with a pointer. In a linked list, a pointer can contain information about connection to the next or previous data. A linked list can be expressed as a singly linked list, a doubly linked list, or a circular linked list depending on the form. A stack can be a data enumeration structure that allows limited access to data. A stack can be a linear data structure in which data can be processed (eg, inserted or deleted) at only one end of the data structure. The data stored in the stack may be a LIFO-Last in First Out (Last in First Out) data structure. A queue is a data listing structure that allows limited access to data, and unlike a stack, it can be a data structure (FIFO-First in First Out) in which data stored later comes out later. A deck can be a data structure that can handle data from either end of the data structure.

The nonlinear data structure may be a structure in which a plurality of data are connected after one data. The non-linear data structure may include a graph data structure. A graph data structure can be defined as a vertex and an edge, and an edge can include a line connecting two different vertices. A graph data structure may include a tree data structure. The tree data structure may be a data structure in which one path connects two different vertices among a plurality of vertices included in the tree. That is, it may be a data structure that does not form a loop in a graph data structure.

Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, a neural network is unified and described. The data structure may include a neural network. And the data structure including the neural network may be stored in a computer readable medium. The data structure including the neural network may also include preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation function associated with each node or layer of the neural network, and neural network It may include a loss function for learning of . A data structure including a neural network may include any of the components described above. That is, the data structure including the neural network includes preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation function associated with each node or layer of the neural network, and neural network. It may be configured to include all or any combination thereof, such as a loss function for learning of . In addition to the foregoing configurations, the data structure comprising the neural network may include any other information that determines the characteristics of the neural network. In addition, the data structure may include all types of data used or generated in the computational process of the neural network, but is not limited to the above. A computer readable medium may include a computer readable recording medium and/or a computer readable transmission medium. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network includes one or more nodes.

The data structure may include data input to the neural network. A data structure including data input to the neural network may be stored in a computer readable medium. Data input to the neural network may include training data input during a neural network learning process and/or input data input to a neural network that has been trained. Data input to the neural network may include pre-processed data and/or data subject to pre-processing. Pre-processing may include a data processing process for inputting data to a neural network. Accordingly, the data structure may include data subject to pre-processing and data generated by pre-processing. The foregoing data structure is only an example, and the present disclosure is not limited thereto.

The data structure may include the weights of the neural network. (In this specification, weights and parameters may be used in the same meaning.) Also, a data structure including weights of a neural network may be stored in a computer readable medium. A neural network may include a plurality of weights. The weight may be variable, and may be changed by a user or an algorithm in order to perform a function desired by the neural network. For example, when one or more input nodes are interconnected by respective links to one output node, the output node is set to a link corresponding to values input to input nodes connected to the output node and respective input nodes. A data value output from an output node may be determined based on the weight. The foregoing data structure is only an example, and the present disclosure is not limited thereto.

As a non-limiting example, the weights may include weights that are varied during neural network training and/or weights for which neural network training has been completed. The variable weight in the neural network learning process may include a weight at the time the learning cycle starts and/or a variable weight during the learning cycle. The weights for which neural network learning has been completed may include weights for which learning cycles have been completed. Accordingly, the data structure including the weights of the neural network may include a data structure including weights that are variable during the neural network learning process and/or weights for which neural network learning is completed. Therefore, it is assumed that the above-described weights and/or combinations of weights are included in the data structure including the weights of the neural network. The foregoing data structure is only an example, and the present disclosure is not limited thereto.

The data structure including the weights of the neural network may be stored in a computer readable storage medium (eg, a memory or a hard disk) after going through a serialization process. Serialization can be the process of converting a data structure into a form that can be stored on the same or another computing device and later reconstructed and used. A computing device may serialize data structures to transmit and receive data over a network. The data structure including the weights of the serialized neural network may be reconstructed on the same computing device or another computing device through deserialization. The data structure including the weights of the neural network is not limited to serialization. Furthermore, the data structure including the weights of the neural network is a data structure for increasing the efficiency of operation while minimizing the resource of the computing device (for example, B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree). The foregoing is only an example, and the present disclosure is not limited thereto.

The data structure may include hyper-parameters of the neural network. Also, the data structure including the hyperparameters of the neural network may be stored in a computer readable medium. A hyperparameter may be a variable variable by a user. Hyperparameters include, for example, learning rate, cost function, number of learning cycle iterations, weight initialization (eg, setting the range of weight values to be targeted for weight initialization), hidden unit number (eg, the number of hidden layers and the number of nodes in the hidden layer). The foregoing data structure is only an example, and the present disclosure is not limited thereto.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of exemplary approaches. Based upon design priorities, it is to be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure. The accompanying method claims present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art of this disclosure, and the general principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. Thus, the present disclosure is not to be limited to the embodiments presented herein, but is to be interpreted in the widest scope consistent with the principles and novel features presented herein.

Claims (19)

A computer program stored in a computer-readable storage medium, which, when executed in one or more processors, performs the following operations for generating a document image using at least one network function, the operations comprising:
generating an actualized digital document image by inputting the digital document image to the first generation network;
generating a restored digital document image by inputting the actualized digital document image to a second generation network;
inputting the actualized digital document image and the actual document image to a first identification network to identify a difference in a first image property; and
training the first generation network so that the digital document image and the restored digital document image match, and a difference in the first image property is equal to or less than a predetermined level;
including,
A computer program stored on a computer-readable storage medium.
According to claim 1,
The digital document image is generated based on digital document image creation data including elements for generating a digital document image.
A computer program stored on a computer-readable storage medium.
According to claim 1,
The digital document image is obtained by crawling open digital document data,
A computer program stored on a computer-readable storage medium.
According to claim 1,
The actualized digital document image is an image for which style conversion has been performed on the digital document image by the first generation network, and has a style similar to that of the actual document image;
The restored digital document image is an image for which style conversion has been performed on the actualized digital document image by the second generation network, and has a style similar to that of the digital document image.
A computer program stored on a computer-readable storage medium.
According to claim 1,
The second generating network is in an inverse function relationship with the first generating network and has a network structure that is a mirror image of the first generating network.
A computer program stored on a computer-readable storage medium.
According to claim 1,
The first identification network includes an operation for assigning a probability value indicating whether the actual document image is a real document image to each of the actualized digital document image and the real document image in order to identify the two input document images,
A computer program stored on a computer-readable storage medium.
According to claim 1,
The first identification network is trained using a loss function based on a probability value indicating whether a real document image is a real document image and a probability value indicating whether a real document image is a real document image.
A computer program stored on a computer-readable storage medium.
delete delete delete According to claim 1,
generating a restored real document image by inputting the real document image to a second generation network;
generating a real document image by inputting the restored real document image to the first generation network;
inputting the restored real document image and the digital document image to a second identification network to identify a difference in a second image property; and
learning the second generation network so that the real document image and the actualized real document image match, and a difference in the second image property is equal to or less than a predetermined level;
Including more,
A computer program stored on a computer-readable storage medium.
According to claim 11,
The actual document image is obtained by crawling an actual document image published on the Internet.
A computer program stored on a computer-readable storage medium.
According to claim 11,
The restored real document image is an image for which style conversion has been performed on the real document image by the second generation network, and has a style similar to that of the digital document image;
The actualized real document image is an image for which style conversion has been performed on the restored real document image by the first generation network, and has a style similar to that of the real document image.
A computer program stored on a computer-readable storage medium.
According to claim 11,
The second identification network includes an operation for assigning a probability value indicating whether or not the document image is a digital document image to each of the restored real document image and the digital document image in order to identify the two input document images.
A computer program stored on a computer-readable storage medium.
According to claim 11,
The second identification network is learned using a loss function based on a probability value indicating whether a restored real document image is a digital document image and a probability value indicating whether a digital document image is a digital document image.
A computer program stored on a computer-readable storage medium.
delete A method for generating a document image in which each step is performed by a computing device, comprising:
generating an actualized digital document image by inputting the digital document image to a first generation network;
generating a restored digital document image by inputting the actualized digital document image to a second generation network;
inputting the actualized digital document image and the actual document image into a first identification network to identify a difference in a first image attribute; and
training the first generating network such that the digital document image and the restored digital document image match, and a difference in the first image property is equal to or less than a predetermined level;
including,
A method for generating document images.
In the document image generator,
one or more processors for processing document image data; and
a memory for storing at least one deep learning-based network function;
includes, and
the one or more processors
inputting the digital document image into a first generation network to generate a realized digital document image;
inputting the actualized digital document image to a second generation network to generate a restored digital document image;
inputting the actualized digital document image and the actual document image into a first identification network to identify a difference in a first image property; and
training the first generating network such that the digital document image and the reconstructed digital document image match, and a difference in the first image property is less than or equal to a predetermined level;
Document image generator.
delete

Images