KR102507534B1 - Method and apparatus for accounting management using ocr recognition based on artificial intelligence - Google Patents

Abstract

According to various embodiments, an accounting management server using artificial intelligence-based OCR recognition comprises: a DB management unit which obtains a document image from a user terminal; a character recognition unit which scans the obtained document image and recognizes characters and digits contained in each of document images; a document type prediction unit which predicts a document type of a document corresponding to the obtained document image; and a document accounting analysis unit which, based on the characters and digits recognized by the character recognition unit and the predicted document type, estimates an accounting producing formula for performing accounting on the document image. The document type prediction unit may comprise a document type prediction model which uses pre-trained Gaussian mixture model, clusters the document images corresponding to each of learning data obtained from multiple document images to determine a cluster corresponding to each of the document images and predicts a document type granted to the determined cluster as a document type of a document corresponding to the document image included in the cluster. The purpose of the present invention is to provide a method for predicting types of documents based on obtained images.

Description

Accounting management method and apparatus using artificial intelligence-based OCR recognition

The present invention relates to an accounting management method and apparatus using artificial intelligence-based OCR recognition, and more particularly, recognizes letters and numbers included in a document image, determines the type of document, and calculates accounting according to the type of document. It relates to a method and apparatus for performing accounting by determining an expression.

Unless otherwise indicated herein, material described in this section is not prior art to the claims in this application, and inclusion in this section is not an admission that it is prior art.

Individuals or companies store and manage various documents collected during economic activities as post-documentary documents for accounting processing such as expense reports and expenditure resolutions, or tax processing such as global income tax returns. Since the documents stored and managed in this way are made of paper, there is a fundamental risk of damage to the original, there is a technical and economic burden to prevent exposure to contamination, loss, and corruption, and document storage space is proportional to the increase in the amount of documents stored. There is a problem that needs to be increased.

In addition, the person in charge of an individual or company extracts and classifies the information necessary for the above-mentioned accounting or tax processing from conventional documents in a handwritten input method, records it in a book, or inputs and stores it in a PC where an accounting management program is installed, so information is extracted. There are problems with working.

Meanwhile, the text included in the document image can be converted through encoding. The text converted through encoding can be edited and searched electronically, and the converted text can be stored in a database in the form of a file. This mechanical encoding can be performed mainly through optical character recognition (OCR), a technology that finds the position of a character in an image and finds out what character it is, and automatically detects an image-based text document using a computer, etc. can be identified and encoded.

However, the information collection method according to the prior art has a problem in that the inspector has to inspect the information collected through OCR to determine whether or not there is an abnormality in the recognized information, and when the same OCR recognition code is applied to all document images, different results are obtained. There has been a problem with the Therefore, it is necessary to prepare a method for determining a document type only with characters recognized from a document image scanned or photographed without a document title and recommending an accounting calculation method according to the document type.

An object of the present invention to solve the above problems is to provide a method for predicting a document type based on an acquired image.

Another object of the present invention to solve the above problems is to provide a method for quickly determining a predefined accounting formula according to the type of document and assisting in accounting.

Another object of the present invention for solving the above problems may further provide objects directly or indirectly identified through this document.

According to various embodiments, an accounting management server using artificial intelligence-based OCR recognition includes a DB management unit that acquires a document image from a user terminal; a character recognizing unit that scans the acquired document images and recognizes letters and numbers included in each of the document images; a document type prediction unit that predicts a document type of a document corresponding to the acquired document image; a document accounting analysis unit for estimating an accounting calculation formula for accounting for the document image based on the character, the number, and the predicted document type recognized through the character recognition unit; A cluster corresponding to each of the document images is determined by clustering the document images corresponding to each of training data obtained from a plurality of document images using a pre-learned Gaussian mixture model, and the determined cluster It may include a document type prediction model that predicts a document type assigned to the document type of a document corresponding to a document image belonging to a corresponding cluster.

According to various embodiments of the present disclosure, the text recognition unit converts the document image into a gray scale and performs binarization on the document image converted into a gray scale, but each pixel of pixels included in the document image A value is detected, a first area in which the detected pixel value is lower than a first threshold value, and a second area in which the detected pixel value is higher than a second threshold value which is higher than the first threshold value are determined, and the pixel value of the first area is determined. Image correction may be performed by lowering the value by a difference value between the first threshold value and the second threshold value and by increasing the pixel value of the second area by the difference value.

According to various embodiments of the present disclosure, the character recognition unit detects individual characters and individual numbers based on the preprocessed document image after image correction is performed, calculates a distance between the detected individual characters, and calculates a distance between the individual characters that is less than or equal to a predefined distance. Group them into a string, calculate the distance between the detected individual numbers, group individual numbers that are less than a predefined distance into a string of numbers, and match the string with an accounting item list stored in the DB management unit to correspond to the string. Accounting items can be determined.

According to various embodiments of the present disclosure, the character recognition unit determines a sequence of numbers corresponding to the determined accounting item, and scans the sequence of numbers in a first direction based on the accounting item when the predetermined accounting items are detected. and, as a result of the scanning, when the sequence of numbers is not detected and another character string distinguished from the accounting item is detected, the sequence of numbers is detected by scanning in a second direction based on the accounting item, wherein the sequence of numbers is detected in the first direction. and the second direction may be directions perpendicular to each other.

According to various embodiments of the present disclosure, the document accounting analysis unit performs accounting processing on the document image based on the accounting item and the number sequence corresponding to the accounting item, and types of accounting calculation amounts to which amounts corresponding to the number sequence are to be applied. , input the amounts into an accounting formula to which the amounts are applied, and perform accounting processing on the document image based on an output value of the accounting formula.

According to various embodiments, the document type prediction model is a clustering model that operates based on a Gaussian mixture model, and uses learning data collected for each of the plurality of document images to determine the number of character strings, number sequences, and accounting items. , obtaining pre-processing learning data including a document configuration format identifier, and at least the number of character strings, the number of numeric sequences, the number of accounting items, and fraudulent document configuration format identifiers among indices included in the preprocessing learning data, respectively. Generating a first learning feature vector having as a component value, classifying the document image into one of a plurality of clusters using the first learning feature vector, selecting a Gaussian distribution among K Gaussian distributions of the Gaussian mixture model A Gaussian distribution having the highest result value of the function is selected, the selected Gaussian distribution is determined as a document type of a document corresponding to the document image, and K may be a natural number greater than 1.

According to various embodiments, the document accounting analysis unit may determine a document type corresponding to each of the predicted document images, a training input value corresponding to the learning data, and variables and symbols constituting the accounting formula. and an accounting calculation formula estimation model in which feature values are supervised-learned using training data composed of training output values and the accounting calculation formula is estimated based on deep learning, wherein the accounting calculation formula estimation model includes the accounting calculation formula estimation model. A second learning feature vector obtained by converting feature values related to the number of character strings, the number of numeric strings, the number of accounting items, the document type identification symbol, and the document composition type identification symbol obtained from a plurality of document images as an input value, Supervised learning may be performed in advance using training data having a feature vector obtained by converting feature values of the variables and symbols constituting the accounting formula as an output value.

According to various embodiments of the present disclosure, the character recognition unit identifies a type of language corresponding to the individual character included in the document image, and determines an accounting item corresponding to the character string according to the type of the identified language. When the individual characters included in the string are characters of the first language, the sum of the width and height of the individual characters is within a preset first critical range, and the ratio of the width and height is a preset first ratio. The accounting item corresponding to the character string is determined based on individual characters in the character string, and when the individual characters included in the character string are characters of a second language different from the first language, the width of each character among the individual characters. determining the accounting item corresponding to the character string based on individual characters in which a sum of the height and the width are within a preset second threshold range, and a ratio of the width and the height is within a preset second threshold range; The range may be different from the first threshold range, and the second ratio may be different from the first ratio.

According to various embodiments of the present disclosure, the character recognition unit stores a scan direction in which the number sequence of accounting items scanned prior to the accounting item has been successfully detected, determines a ratio of the scan direction based on the stored scan direction, and A different weight is applied to each of the scanned accounting items, and a scan direction corresponding to the accounting item is determined based on the determined ratio of the scan direction and the weight, and a scan index is calculated based on the following equation,

In the above equation, S is the scan index, r _n is a weight for the n-th accounting item in which the number sequence is detected by scanning in the first direction, d _m- is m in which the number sequence is detected by scanning in the second direction Means a weight for the th accounting item, r _n- increases as the distance between the n th accounting item and the standard accounting item becomes shorter, and increases as n increases, and d _m is the distance between the m th accounting item and the standard accounting item increases as m becomes shorter and increases as m increases, the distance between the accounting items is calculated based on the coordinates of the central point of the boundary box surrounding the accounting items, and when the calculated scan index is greater than or equal to a preset critical index, the first Scanning may be started first in the direction, and when the calculated scan index is smaller than a preset threshold index, scanning may be started first in the second direction.

According to various embodiments disclosed in this document, a document type may be known through a document image itself without a title of the document image.

Also, according to various embodiments, a document type of a document image may be labeled and an accounting formula may be rapidly derived based on the labeled training data.

In addition to this, various effects identified directly or indirectly through this document may be provided.

1 is a diagram illustrating an accounting management system according to an embodiment.
Figure 2 is a diagram showing the main components of the accounting management server.
3 is a diagram for determining a document type of a document corresponding to a text image through a clustering model.
4 is a method for generating a feature vector to be input to an accounting formula estimation model including an identification symbol indicating the document type when the document type of a document image is determined, and estimating an accounting formula to be applied in response to the document image through the feature vector. It is a drawing about
5 is a flowchart illustrating a process of reviewing a sequence of numbers corresponding to an accounting item according to an exemplary embodiment.
6 is a diagram showing the hardware configuration of the accounting management server according to FIG. 1 .

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Like reference numerals have been used for like elements throughout the description of each figure.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The terms and/or include any combination of a plurality of related recited items or any of a plurality of related recited items.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating an accounting management system 10 in one embodiment. Referring to FIG. 1 , the accounting management system 10 may include an accounting management server 100 and a user terminal 200 . Operations described below may be performed through a platform (eg, a web page and/or a kind of application) controlled by the accounting management server 100 .

The accounting management server 100 may obtain a document image from the user terminal 200, read the acquired document image, and predict a document type of a document corresponding to the document image. After predicting the document type of the document corresponding to the document image, the accounting management server 100 determines an accounting formula to be applied based on the predicted document type and letters and/or numbers detected in the document image, Accounting can be performed by entering an accounting item and a number corresponding to the accounting item in the accounting formula.

The user terminal 200 includes a communicable desktop computer, a laptop computer, a notebook, a smart phone, a tablet PC, a mobile phone, and a smart watch. (smart watch), smart glass, e-book reader, PMP (portable multimedia player), portable game console, navigation device, digital camera, DMB (digital multimedia broadcasting) player, digital voice It may be a digital audio recorder, a digital audio player, a digital video recorder, a digital video player, and a personal digital assistant (PDA).

The accounting management server 100 and the user terminal 200 are connected to the communication network 50, respectively, and can transmit and receive data to and from each other through the communication network 50. For example, the communication network 50 may include a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Global System for Mobile Network (GSM), an Enhanced Data GSM Environment (EDGE), a High Speed Downlink Packet Access (HSDPA), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Zigbee, Wi-Fi, Voice over VoIP Internet Protocol), LTE Advanced, IEEE802.16m, WirelessMAN-Advanced, HSPA+, 3GPP Long Term Evolution (LTE), Mobile WiMAX (IEEE 802.16e), UMB (formerly EV-DO Rev. C), Flash-OFDM, iBurst and Various types of wired or wireless networks can be used, such as MBWA (IEEE 802.20) systems, HIPERMAN, Beam-Division Multiple Access (BDMA), World Interoperability for Microwave Access (Wi-MAX), and 5G.

Figure 2 is a diagram showing the main components of the accounting management server (100). The accounting management server 100 may include a DB management unit 101, a text recognition unit 102, a document type prediction unit 103, a document accounting analysis unit 104, and the like.

The DB management unit 101 may acquire a plurality of document images from the user terminal 200 . The document images may include images transmitted from the user terminal 200 and/or an external server connected through a network and/or images scanned through a scanner. The DB management unit 101 may store characters recognized or detected through the character recognition unit 102 as OCR data. For example, the DB management unit 101 may store character strings and/or numeric strings detected in document images as OCR data.

The DB management unit 101 may store recognition items for each document type. Recognition items may refer to essential or important items required in documents for each document type. Recognition items may be stored not only in the national language, but also in the format of other languages. For example, the recognized items in the lease agreement are the lessor's name, the lessee's name, the lessee's business registration number, the lessee's resident registration number, the lessee's contact information, deposit, monthly rent, monthly rent payment date, monthly maintenance fee, monthly maintenance fee payment date, contract start date, contract It may include items related to the end date, etc. For example, items recognized in an outsourcing contract may include items related to trade name, business registration number, contract start date, contract end date, monthly service fee, payment date, and the like. For example, items recognized in a tax invoice may include items related to trade name, business registration number, preparation date, supply value, tax amount, and the like. For example, the recognition items in the simple receipt may include items related to a trade name, business registration number, date of creation, amount, and the like. For example, the recognition item in the utility bill may include items related to the type of utility bill, issuing institution, notice amount, payment due date, payment account, payment method, and the like.

The DB management unit 101 may store accounting items for each document type. Accounting items refer to items required for accounting calculation, and may be some of the above recognition items. Accounting items can be saved not only in the Korean language, but also in the format of other languages. An accounting item may be an item related to a specific number or amount. For example, the accounting items in the lease contract may include items related to monthly rent, monthly rent payment date, monthly management fee, monthly management fee payment date, contract start date, contract end date, and the like. For example, accounting items in an outsourcing contract may include items related to a contract start date, a contract end date, a monthly service fee, and a payment date. For example, items recognized in a tax invoice may include items related to supply value and tax amount. For example, accounting items in a receipt may include items related to supply value for each item, value-added tax, and the like. For example, accounting items in a utility bill may include items related to a bill amount, payment due date, and the like.

The character recognition unit 102 may pre-process the image by correcting and normalizing the document image in order to detect characters and/or numbers through the input document image. For example, the text recognition unit 102 converts the document image into a gray scale, and the text recognition unit 102 performs binarization on the document image converted to gray scale. A pixel value of each included pixel is detected, and a first area in which the detected pixel value is lower than a preset first threshold value and a second area higher than a preset second threshold value higher than the first threshold value are selected. can decide The character recognition unit 102 performs image correction by lowering the pixel value of the first area by a difference value between the first threshold value and the second threshold value and by increasing the pixel value of the second area by the difference value. can be done

The character recognition unit 102 may detect individual characters and/or individual numbers based on the preprocessed document image after image correction is performed. For example, individual characters may mean syllable components based on Korean, and individual alphabets based on English.

The character recognition unit 102 may detect individual characters and/or individual numbers, calculate a distance between the detected individual characters, and group adjacent individual characters into a character string. The character recognition unit 102 may group adjacent individual numbers into a number sequence by calculating a distance between the detected individual numbers. The text recognition unit 102 may register the grouped character string as a valid character string and register the grouped number sequence as a valid number sequence.

The text recognition unit 102 may create a boundary box including a valid character string and/or numeric string, and determine coordinates of four vertices of the created boundary box. The text recognition unit 102 may calculate the tilt angle of the boundary box based on the determined coordinates of the four vertexes of the boundary box. The character recognition unit 102 may recognize or identify the character string and/or the number sequence by rotating it by the calculated angle. In other words, the character recognition unit 102 may recognize or identify a character string and/or a number sequence by correcting only the tilted portion in order to accurately read the document image.

The character recognition unit 102 may identify a language type corresponding to each character included in the document image, and determine an accounting item corresponding to the character string according to the identified language type. For example, if the individual characters included in the character string are characters of the first language, the character recognition unit 102 determines that the sum of the width and height of the individual characters among the detected individual characters is within a preset first threshold range. , An accounting item corresponding to the character string may be determined based on individual characters in which the ratio of the width to the height is within a preset first ratio.

The character recognition unit 102 determines that when the detected individual characters are characters of a second language different from the first language, the sum of the width and height of the individual characters among the detected individual characters is within a preset second threshold range, An accounting item corresponding to the character string may be determined based on individual characters in which the ratio of the width to the height is within a preset second ratio. The second threshold range may be different from the first threshold range, and the second ratio may be different from the first ratio.

The character recognition unit 102 may determine amounts corresponding to each of predetermined accounting items as necessary for accounting management. When the predetermined accounting item is detected, the character recognition unit 102 detects a sequence of numbers by scanning in a first direction based on the accounting item, and as a result of the scan, the sequence of numbers is not detected and is distinguished from the accounting item. When another accounting item is detected, a sequence of numbers may be detected by scanning in the second direction based on the accounting item. The text recognition unit 102 may determine the detected number sequence as an amount corresponding to an accounting item.

The text recognition unit 102 may determine a scan direction corresponding to an accounting item based on a scan direction corresponding to an accounting item scanned prior to the accounting item. The text recognition unit 102 may record the scanning direction in which the number sequence of the accounting items scanned prior to the accounting item has been successfully detected, and determine the ratio of the scanning direction. The character recognition unit 102 may determine a scan direction corresponding to the accounting item based on the ratio of the determined scan direction. For example, when the character recognition unit 102 succeeds in detecting a sequence of numbers in the first direction, second direction, and second direction in order from the previously scanned accounting items, the sequence of numbers when scanned in the second direction Since the detection rate is high, the scan direction corresponding to the accounting item may be determined as the second direction.

The character recognition unit 102 may determine a scan direction corresponding to an accounting item by giving different weights to each previous scan direction (or an accounting item corresponding to the scan direction) rather than simply considering the number of previous scan directions. there is. When scanning the current accounting item, since the configuration of the accounting item scanned immediately before may be most similar to the configuration of the current accounting item, different weights may be assigned to each previous scan direction. Current accounting items may be referred to as base accounting items.

The character recognition unit 102 calculates a scan index based on previous scan directions, and when the calculated scan index is equal to or greater than a preset threshold index, first starts scanning in the first direction, and the calculated scan index is set in advance. If it is smaller than the critical index, scanning may be started first in the second direction. This has the effect of more efficiently and quickly determining a number string (eg, amount) corresponding to an accounting item.

The text recognition unit 102 may calculate the scan index based on Equation 1 below.

In Equation 1, S is the scan index, r _n is the weight for the n-th accounting item in which the number sequence is detected by scanning in the first direction, and d _m- is the m-th number sequence in which the number sequence is detected by scanning in the second direction. Means a weight for an accounting item, r _n- increases as the distance between the n-th accounting item and the standard accounting item becomes shorter, and increases as n increases, and d _m is the distance between the m-th accounting item and the standard accounting item The shorter it is, the larger it is, and the larger m is, the larger it can be. The distance between the accounting items may be calculated based on coordinates of a central point of a boundary box surrounding the accounting items.

The document type prediction unit 103 may predict document types of documents corresponding to each of the document images by using a plurality of document images. The document type predictor 103 may identify a document composition format based on the document image obtained from the user terminal 200 . The document type prediction unit 103 may identify whether the document image is configured in a table (table) format, whether the document is configured in a table of contents format without a table, or whether the document is configured in a letter (letter) format, and the like. . The document type prediction unit 103 may predict the document type of the document based on a document image and/or characters/numbers detected in the document image, positions of the characters/numbers, and a document configuration type. For example, the document type may include a lease contract, an outsourcing contract, a tax invoice, a receipt, a simple receipt, an invoice, a utility bill, a transaction statement, and the like. When the document type of the document corresponding to the document image is predicted through the document type prediction unit 103, the user does not have to directly input the document type, and document accounting analysis according to the document proceeds with the document type being determined. Accurate accounting is possible.

The document type prediction unit 103 may predict a document type of a document corresponding to a document image based on training data according to a document image. The document type prediction unit 103 determines a cluster corresponding to a document image by using a document type prediction model (eg, a clustering model), and converts a document type assigned to the determined cluster into a document type of a document corresponding to the document image. Predictable. Since the clustering model is implemented using a machine learning-based Gaussian mixture model, accounting formula estimation accuracy according to document images can be implemented relatively accurately by using the output value labeled through the clustering model for the accounting formula estimation model.

The document accounting analysis unit 104 may perform accounting processing on the document image based on an accounting item determined in the document image and a sequence of numbers corresponding to the accounting item.

The document accounting analysis unit 104 may determine the type of accounting calculation amount to which amounts corresponding to the number sequence are applied. The document accounting analysis unit 104 may input the amounts into an accounting formula to which the amounts are applied. The document accounting analysis unit 104 may perform accounting for the document image based on the output value of the accounting formula.

3 is a diagram for determining a document type of a document corresponding to a text image through a clustering model.

The document type prediction model may include a clustering model that operates based on a Gaussian Mixture Model (GMM). The clustering model may correspond to a kind of source code that implements a Gaussian Mixture Model (GMM) in software or an operation program of the source code. Since the types of accounting items and the number of accounting items detected are different for each document image, even if a specific accounting item is included, it may not be the same document type. For example, even if the document image includes the same accounting item called down payment, the types of documents including the accounting item called down payment can be extremely diverse. Therefore, it is necessary to predict the document type based on the type and number of accounting items included in each document image.

In an embodiment of the present invention, in consideration of the fact that the document type is different for each document image, the most suitable accounting formula is estimated according to the document type, and the accounting calculation is performed based on the estimated accounting formula. A plurality of clusters are formed by clustering the images, and a document type corresponding to each of the formed clusters is determined. Here, the document type is a kind of identification symbol individually assigned to each cluster to which the document image belongs, and may be expressed as an identification number determined without superiority or inferiority, such as 1, 2, 3, etc., or may be expressed in various types of unique identification symbols. there is.

When the document type corresponding to each of the document images is determined through clustering, the document accounting analysis unit 104 classifies learning data for the document images according to the document type and pre-processes the vector to be input to the accounting formula estimation model. values for can be determined.

The clustering model may generate a learning feature vector using training data collected for each document image. For example, the clustering model may obtain pre-processing learning data including the number of character strings, the number of numeric strings, the number of accounting items, and the document configuration format identification symbol using training data collected for each document image. The clustering model may generate a first learning feature vector corresponding to a user by using at least some of indicators included in the obtained preprocessing learning data. For example, the clustering model may generate a first learning feature vector having each of the number of character strings detected or recognized through a document image, the number of numeric strings, the number of accounting items, and the document composition type identification symbol as component values. .

The clustering model may classify each of the document images into one of a plurality of clusters using the first training feature vector of each of the document images. For example, in the case of a Gaussian Mixture Model (GMM), each of a predetermined K (K is a natural number greater than 1) Gaussian distributions corresponds to a cluster. It may operate to classify document images into one of K Gaussian distributions.

Specifically, in the case of the clustering model, a Gaussian distribution having the highest result value of the Gaussian distribution selection function (γ) according to Equation 2 below is selected among K Gaussian distributions using Bayes' theorem, The selected Gaussian distribution may be determined as a document type of a document corresponding to a corresponding document image.

In Equation 2, (γ) is a selection function, x _n is the first learning feature vector according to the nth document image, and z _nk is k in the Gaussian mixture model when the first learning feature vector is given (k is 1 and K)th Gaussian distribution is selected and is 1, otherwise it is a binary variable with a value of 0. Also, μ and Σ in Equation 2 are parameters according to the Gaussian mixture model, and are values determined through a learning process for the Gaussian mixture model in advance. The learning process for determining the parameters according to the Gaussian mixture model can be understood by referring to Christopher Bishop's Pattern Recognition and Machine Learning, 2008.03.18, so a detailed description is omitted.

4 is a method for generating a feature vector to be input to an accounting formula estimation model including an identification symbol indicating the document type when the document type of a document image is determined, and estimating an accounting formula to be applied in response to the document image through the feature vector. It is a drawing about

The document accounting analysis unit 104 may determine or estimate the accounting formula by using an accounting formula estimation model. The accounting formula estimation model is a second learning feature vector obtained by converting feature values related to the number of character strings, the number of numeric strings, the number of accounting items, the document type, and the document composition type identification symbol obtained from a plurality of document images. Supervised learning may be performed in advance using training data having as an input value and a feature vector obtained by converting feature values of variables and symbols constituting the accounting formula as an output value.

The document accounting analysis unit 104 determines the number of character strings, number strings, number of accounting items, document type identifiers, document composition format identifiers, etc. obtained from the document image in the supervised accounting formula estimation model. Characteristic values are converted into input vectors and inputted to the accounting formula estimation model, and feature values related to variables and symbols constituting the accounting formula are obtained by converting an output vector obtained as an output of the accounting formula estimation model. And, the accounting formula can be estimated using the feature values of the variables and symbols.

In other words, the accounting calculation formula estimation model, the input document image and / or document composition format, document type, the number of accounting items included in the document image and / or document, the number of character strings, the number of numeric strings, and accounting items A predefined loss using training data that takes as input values characteristic values of training data including coordinate values indicating the location of , and outputs identification symbols indicating the document type of the document image. It can be supervised so that the output value of the function is minimized.

For example, the configuration format may include a table format, a letter format, a table of contents format, and the like, and the accounting item is a business registration number, deposit, monthly rent, contract start date, contract end date, and month Items such as service fee, payment date, supply value, tax amount, etc. can be included. The coordinate value indicating the position of the accounting item may mean a coordinate value (eg, a pixel position value) corresponding to a central point of a boundary box including the accounting item.

The deep learning-based accounting formula estimation model, when the second learning feature vectors for each document image are sequentially input as training input values constituting the training data, the document type obtained as the output value of the accounting calculation estimation model is combined with the training output value. It is supervised learning by comparing each other and tuning the parameters constituting the accounting formula estimation model according to the comparison result. Specifically, the deep learning-based accounting formula estimation model uses a predefined loss using an output vector indicating the document type obtained as an output value of the accounting formula estimation model and a training output vector indicating the document type according to the training output value. It is calculated according to a loss function, and parameters constituting an accounting formula estimation model may be adjusted so that a resultant value of the calculated loss function is minimized. In this case, the loss function may be a cross entropy function.

The accounting formula estimation model receives a second learning feature vector (X), and is transmitted from an input layer composed of input nodes of the same number (N) as the number of component values of the second learning feature vector input, and the input layer. Determine the probability (p) corresponding to the output vector (Y`) by applying an activation function to the hidden layer and the output vector (Y`) that delivers the output vector (Y`) calculated using the received output values to the output layer, An output layer outputting an output vector Y′ having the highest determined probability p may be included. In the present invention, each of the nodes constituting the accounting formula estimation model may be interchangeably referred to as a neuron, which is an expression commonly used in the technical field to which the present invention belongs.

Specifically, the accounting formula guess model receives the second learning feature vector (X) provided as a training input value, the output vector (Y`) obtained as an output of the hidden layer and the document type provided as the training output value Indicating A loss function is calculated using the training output vector (Y), and supervised learning is performed so that the resulting value of the calculated loss function is minimized.

For example, the loss function H(Y,Y′) may be a cross entropy function. The cross entropy (H(Y,Y`)) between the output vector (Y`) and the training output vector (Y) can be defined as in Equation 3 below.

In Equation 3, Ym is the m-th component (m is a natural number greater than or equal to 1) of the training output vector Y, and Y'm may be the m-th component of the output vector Y'.

5 is a flowchart illustrating a process of reviewing a sequence of numbers corresponding to an accounting item according to an exemplary embodiment.

The DB management unit 101 may obtain a photographed document image and store the acquired document image (S510).

The character recognition unit 102 may recognize characters included in the document image using artificial intelligence-based OCR (S520). The character recognition unit 102 may detect a character string and a number sequence in a document image, and may identify an accounting item based on the detected character string and number sequence. The text recognition unit 102 may generate a boundary box to include accounting items for each accounting item, and recognize the position of the accounting item based on the coordinate values of the boundary box.

The character recognition unit 102 may recognize the location of a recognition item (eg, an accounting item) based on the item allowable value biz rule (S530). The character recognition unit 102 may recognize an item value (eg, a number sequence, an amount) corresponding to a recognition item (eg, an accounting item) (S540).

The document accounting analysis unit 104 matches the first amount, which is the amount for each accounting item included in the document image captured by the user terminal 200, and the second amount, which is the amount for each accounting item recognized through the text recognition unit 102. A confirmation request may be made (S550).

If the user finds an error in which the first amount and the second amount are different in step S550, the document accounting analysis unit 104 is an item that needs to be corrected based on the information about correction matters obtained from the user terminal 200. It is possible to modify the item value for (S560).

When the document accounting analysis unit 104 obtains an answer signal indicating that the first amount and the second amount match from the user terminal 200, the item value may be stored in the DB management unit 101 (S570).

FIG. 6 is a diagram showing the hardware configuration of the accounting management server 100 according to FIG. 1 .

Referring to FIG. 6, the accounting management server 100 stores at least one processor 110 and instructions instructing the at least one processor 110 to perform at least one operation. May contain memory.

The at least one operation may include at least some of the above-described operations or functions of the accounting management server 100 and may be implemented in the form of instructions and performed by the processor 110 .

Here, the at least one processor 110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor for performing methods according to embodiments of the present invention. can Each of the memory 120 and the storage device 160 may include at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory 120 may be one of a read only memory (ROM) and a random access memory (RAM), and the storage device 160 may be a flash-memory. , a hard disk drive (HDD), a solid state drive (SSD), or various memory cards (eg, a micro SD card).

In addition, the accounting management server 100 may include a transceiver 130 that performs communication through a wireless network. In addition, the server 100 may further include an input interface device 140, an output interface device 150, a storage device 160, and the like. Each component included in the accounting management server 100 is connected by a bus 170 to communicate with each other. In FIG. 6, the accounting management server 100 has been described as an example, but is not limited thereto. For example, a plurality of user terminals may include a component according to FIG. 6 .

The methods according to the present invention may be implemented in the form of program instructions that can be executed by various computer means and recorded on a computer readable medium. Computer readable media may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on a computer readable medium may be specially designed and configured for the present invention or may be known and usable to those skilled in computer software.

Examples of computer readable media may include hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language codes generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter and the like. The hardware device described above may be configured to operate with at least one software module to perform the operations of the present invention, and vice versa.

In addition, the above-described method or device may be implemented by combining all or some of its components or functions, or may be implemented separately.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

100: accounting management server 200: user terminal

Claims (5)

In the accounting management server using artificial intelligence-based OCR recognition,
DB management unit for obtaining a document image from the user terminal;
a character recognizing unit that scans the acquired document images and recognizes letters and numbers included in each of the document images;
a document type prediction unit that predicts a document type of a document corresponding to the acquired document image;
a document accounting analyzer for estimating an accounting calculation formula for accounting for the document image based on the character, the number, and the predicted document type recognized through the character recognition unit;
The document type prediction unit,
A cluster corresponding to each of the document images is determined by clustering the document images corresponding to each of training data obtained from a plurality of document images using a pre-learned Gaussian mixture model, and the determined cluster A document type prediction model predicting a document type assigned to a document type corresponding to a document image belonging to a corresponding cluster;
The text recognition unit,
Converting the document image to gray scale;
Binarization is performed on the document image converted to gray scale, a pixel value of each of pixels included in the document image is detected, and a first area in which the detected pixel value is lower than a first threshold value and the first Determining a second region higher than a second threshold value that is higher than the threshold value;
and performs image correction by lowering a pixel value of the first area by a difference value between the first threshold value and the second threshold value and by increasing a pixel value of the second area by the difference value. delete In claim 1,
The text recognition unit,
Image correction is performed to detect individual characters and individual numbers based on the preprocessed document image;
By calculating the distance between the detected individual characters, individual characters that are less than a predefined distance are grouped into a string, and the individual numbers that are less than the predefined distance are grouped into a number string by calculating the distance between the detected individual numbers,
Accounting management server for determining an accounting item corresponding to the string by matching the string with the accounting item list stored in the DB management unit. In claim 3,
The text recognition unit,
Determine a sequence of numbers corresponding to the determined accounting item,
When predetermined accounting items are detected, detecting the sequence of numbers by scanning in a first direction based on the accounting items;
As a result of the scanning, when the number sequence is not detected and another character string distinguished from the accounting item is detected, the number sequence is detected by scanning in a second direction based on the accounting item, wherein the number sequence is detected in the first direction and the first direction. Accounting server, where the two directions are perpendicular to each other. In claim 4,
The document accounting analysis unit,
Perform accounting processing on the document image based on the accounting item and the sequence of numbers corresponding to the accounting item;
Determine the type of accounting calculation to which amounts corresponding to the sequence of numbers will be applied;
and inputting the amounts into an accounting formula to which the amounts are applied, and performing accounting processing on the document image based on an output value of the accounting formula.

Images