KR20240101388A - Method and system for distributing model set associated with character recognition - Google Patents

Abstract

The present disclosure provides a method for distributing a model set associated with character recognition performed by at least one processor. This method includes generating a plurality of model sets, determining a target model set to be distributed among the generated plurality of model sets, and distributing the determined target model set to a target system, and generating a plurality of model sets. Each of the sets may include at least one machine learning model configured to recognize characters from document images.

Description

Method and system for distributing model sets associated with character recognition {METHOD AND SYSTEM FOR DISTRIBUTING MODEL SET ASSOCIATED WITH CHARACTER RECOGNITION}

This disclosure relates to a method for distributing a model set, and specifically to a method and system for distributing a model set for performing character recognition from document images.

Optical Character Recognition technology radiates light onto printed matter and recognizes characters based on quantitative differences in the light rays generated from the printed matter. This optical character recognition is used to recognize barcodes attached to products.

Meanwhile, as smartphones became popular, technology that analyzed images captured on smartphones and recognized characters within the images also emerged. Typically, when a business card is photographed through a smartphone, data such as name and phone number are recognized from the business card image, and an application is developed and used in which the recognized name and phone number are automatically entered into an address book.

Additionally, a machine learning model was developed to recognize characters from images. When recognizing characters from images through a machine learning model, the accuracy of character recognition can be improved.

However, the performance of machine learning models may vary depending on the operating environment, such as the type of input data and computing resources. For example, in a first computing system, a first machine learning model may demonstrate optimal performance, but in a second computing system, a second machine learning model may demonstrate improved performance than the first machine learning model.

Accordingly, there is a demand for technology that can deploy machine learning models suitable for the computing system to be deployed.

The present disclosure provides a method for distributing a model set associated with character recognition, a computer program stored in a computer-readable recording medium, a computer-readable recording medium, and a device (system) to solve the above problems.

The present disclosure may be implemented in various ways, including as a method, device (system), and/or a computer program stored in a computer-readable storage medium.

According to an embodiment of the present disclosure, a method for distributing a model set associated with character recognition performed by at least one processor includes generating a plurality of model sets, a target model set to be distributed among the plurality of generated model sets. It includes determining and distributing the determined target model set to the target system, and each of the plurality of model sets may include at least one machine learning model configured to recognize characters from document images.

In addition, each of the plurality of model sets recognizes a plurality of characters from a first machine learning model configured to detect and output a plurality of regions in which characters are expressed in a document image and a plurality of regions output from the first machine learning model. It may include a second machine learning model configured to output.

Additionally, each of the plurality of model sets includes a third machine learning model configured to parse a plurality of characters output from the second machine learning model based on the plurality of entities and output a plurality of target texts associated with the plurality of entities. More may be included.

In addition, the step of distributing the target model set to the target system includes allocating computing resources for the target model set, and the target system performs at least one machine learning program included in the target model set using the allocated computing resources. It can be configured to operate the model.

In addition, in the model set distribution method, after the step of distributing the target model set to the target system, a plurality of document images are applied as the target model set, and a plurality of recognition results are output from the target model set based on the plurality of document images. It may further include a step of monitoring the status of the target model set.

In addition, the step of distributing the target model set to the target system includes determining a monitoring period for the target model set, and the step of monitoring the state of the target model set includes monitoring the state of the target model set during the determined monitoring period. It may include a monitoring step.

In addition, the step of distributing the target model set to the target system includes determining the ratio of document images input to the target model set, and the number of document images corresponding to the determined ratio among all document images is transferred to the target model set. can be entered.

In addition, the target model set includes a first machine learning model and a second machine learning model, and the step of monitoring the state of the target model set includes monitoring the state of the first machine learning model and the state of the second machine learning model. May include steps.

Additionally, each of the plurality of model sets may be different in at least one of a data set used for learning, a learning method, a character set, or an entity set.

In addition, the model set distribution method includes calculating recognition accuracy for each of the plurality of model sets based on data output from each of the plurality of model sets and correct answer data, and calculating recognition accuracy for each of the calculated plurality of model sets. A step of outputting recognition accuracy may be further included.

In addition, the model set distribution method includes calculating a confidence score for each of the plurality of model sets based on state data for each of the plurality of model sets, and calculating a confidence score for each of the calculated plurality of model sets. A step of outputting a trust score may be further included.

Additionally, determining the target model set may include performing a test on each of the plurality of model sets and determining a target model set from among the plurality of model sets based on the performed test.

In addition, the model set distribution method includes, after determining the target model set, applying a document image as the target model set to obtain recognition result data based on the document image, and using the document image and the recognition result data for learning. It may further include generating data and applying the generated training data to a target model set to perform additional learning on at least one machine learning model included in the target model set.

In addition, the target model set includes a machine learning model configured to detect a plurality of sub-areas in which characters are expressed from the document image, and the step of generating training data includes providing a plurality of sub-areas detected from the machine learning model. Outputting a user interface for changing at least one of the plurality of sub-areas through the user interface, and generating learning data that determines the changed sub-area as correct answer data based on the user input. It can be included.

In addition, the target model set includes a machine learning model configured to recognize a plurality of characters from a document image, and the step of generating training data includes outputting a user interface for providing a plurality of characters recognized from the machine learning model. It may include receiving a user input to change at least one of the plurality of characters through a user interface, and generating learning data that determines the changed character as correct answer data based on the user input.

In addition, the target model set includes a machine learning model configured to recognize a plurality of target texts from a document image, and the step of generating training data includes a user interface for providing a plurality of target texts recognized from the machine learning model. It may include outputting, receiving a user input to change at least one of a plurality of target texts through a user interface, and generating learning data that determines the changed target text as correct answer data based on the user input. .

In addition, a document image is output in the first area of the user interface, a plurality of target texts related to a plurality of entities are output in the second area of the user interface, and when a specific target text is selected among the plurality of target texts, the selected specific target At least one character associated with the text may be visualized through a specific graphic element in the first area.

Additionally, determining the target model set may include determining the target model set from among the target model sets generated based on the document type.

In order to execute the above-described model set distribution method on a computer, a computer program stored in a computer-readable recording medium may be provided.

According to an embodiment of the present disclosure, an information processing system includes a memory and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, and the at least one program includes, Generating a plurality of model sets, determining a target model set to be distributed among the generated plurality of model sets, and distributing the determined target model set to the target system; each of the plurality of model sets is a document It may include at least one machine learning model configured to recognize characters from images.

According to some embodiments of the present disclosure, a test may be performed on a plurality of model sets, and a target model set to be distributed to the target system may be determined based on the test results. Accordingly, character recognition can be performed stably and quickly through the model set provided as the target system.

According to some embodiments of the present disclosure, a deployed model set may be monitored for a specified period of time to verify performance for the model set.

According to some embodiments of the present disclosure, the status of each of a plurality of machine learning models included in the model set is monitored, so that machine learning models in which performance in the character recognition process deteriorates can be accurately identified.

According to some embodiments of the present disclosure, various statistics about the state of a model set may be provided. Additionally, information comparing multiple model sets may be provided. Accordingly, based on information and statistical information, the state of the model set can be identified in depth. Through this, areas for improvement in the model set can be accurately identified.

According to some embodiments of the present disclosure, recognition accuracy and/or performance indicators for a model set may be calculated and output. Based on recognition accuracy and/or performance indicators, a set of models may be reasonably selected and used.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned are clear to a person skilled in the art (referred to as a ‘person skilled in the art’) in the technical field to which this disclosure pertains from the description of the claims. It will be understandable.

Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, in which like reference numerals indicate like elements, but are not limited thereto.
1 is a schematic diagram illustrating a process for selecting a model set to be distributed according to an embodiment of the present disclosure.
Figure 2 is a schematic diagram showing a configuration connected to enable communication between an information processing system and a plurality of communication terminals according to an embodiment of the present disclosure.
Figure 3 is a block diagram showing the internal configuration of a communication terminal and an information processing system according to an embodiment of the present disclosure.
Figure 4 is an example diagram for explaining how a machine learning model included in a model set according to an embodiment of the present disclosure is learned.
Figure 5 is an exemplary diagram showing an artificial neural network model according to an embodiment of the present disclosure.
6 is a diagram illustrating a user interface associated with learning a model set according to an embodiment of the present disclosure.
7 is a diagram illustrating a user interface including detailed information about a model set according to an embodiment of the present disclosure.
8 is a diagram illustrating a user interface for distributing a model set according to an embodiment of the present disclosure.
9 is a diagram illustrating a user interface for monitoring the status of a new model set according to an embodiment of the present disclosure.
10 is a diagram illustrating a user interface including trust scores of a distributed model set according to an embodiment of the present disclosure.
FIG. 11 is a diagram illustrating a user interface including error information for a model set according to an embodiment of the present disclosure.
12 is a diagram illustrating a user interface including resource group status for a model set according to an embodiment of the present disclosure.
Figure 13 is a diagram illustrating a user interface associated with a recognition result according to an embodiment of the present disclosure.
Figure 14 is a flowchart illustrating a method for distributing a model set according to an embodiment of the present disclosure.

Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, detailed descriptions of well-known functions or configurations will be omitted if there is a risk of unnecessarily obscuring the gist of the present disclosure.

In the accompanying drawings, identical or corresponding components are given the same reference numerals. Additionally, in the description of the following embodiments, overlapping descriptions of identical or corresponding components may be omitted. However, even if descriptions of components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments and methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the present disclosure is complete and that the present disclosure does not convey the scope of the invention to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification are general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but this may vary depending on the intention or precedent of a technician working in the related field, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Accordingly, the terms used in this disclosure should be defined based on the meaning of the term and the overall content of the present disclosure, rather than simply the name of the term.

In this specification, singular expressions include plural expressions, unless the context clearly specifies the singular. Additionally, plural expressions include singular expressions, unless the context clearly specifies plural expressions. When it is said that a certain part includes a certain element throughout the specification, this does not mean excluding other elements, but may further include other elements, unless specifically stated to the contrary.

Additionally, the term 'module' or 'unit' used in the specification refers to a software or hardware component, and the 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not limited to software or hardware. A 'module' or 'unit' may be configured to reside on an addressable storage medium and may be configured to run on one or more processors. Thus, as an example, a 'module' or 'part' refers to components such as software components, object-oriented software components, class components and task components, processes, functions and properties. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. Components and 'modules' or 'parts' may be combined into smaller components and 'modules' or 'parts' or further components and 'modules' or 'parts'. Could be further separated.

According to an embodiment of the present disclosure, a 'module' or 'unit' may be implemented with a processor and memory, and may be implemented with a circuit (circuitry). Terms such as 'circuit (circuitry)' can refer to a circuit in hardware, but can also refer to a circuit in software. 'Processor' should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. In some contexts, 'processor' may refer to an application-specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), etc. 'Processor' refers to a combination of processing devices, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such combination of configurations. You may. Additionally, 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. 'Memory' refers to random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), May also refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or marking data storage, registers, etc. A memory is said to be in electronic communication with a processor if the processor can read information from and/or write information to the memory. The memory integrated into the processor is in electronic communication with the processor.

According to an embodiment of the present disclosure, a 'module' or 'unit' may be implemented with a processor and memory. 'Processor' should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. In some contexts, 'processor' may refer to an application-specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), etc. 'Processor' refers to a combination of processing devices, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such combination of configurations. You may. Additionally, 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. 'Memory' refers to random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), May also refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or marking data storage, registers, etc. A memory is said to be in electronic communication with a processor if the processor can read information from and/or write information to the memory. The memory integrated into the processor is in electronic communication with the processor.

In the present disclosure, 'system' may include at least one of a server device and a cloud device, but is not limited thereto. For example, a system may consist of one or more server devices. As another example, a system may consist of one or more cloud devices. As another example, the system may be operated with a server device and a cloud device configured together.

In addition, terms such as first, second, A, B, (a), (b) used in the following embodiments are only used to distinguish one component from another component, and the terms The nature, sequence, or order of the relevant components are not limited.

Additionally, in the following embodiments, when a component is described as being 'connected', 'coupled' or 'connected' to another component, the component may be directly connected or connected to the other component. , it should be understood that another component may be 'connected', 'combined', or 'connected' between each component.

In addition, as used in the following embodiments, 'comprises' and/or 'comprising' means that the mentioned component, step, operation, and/or element includes one or more other components, steps, or operations. and/or the presence or addition of elements.

In the present disclosure, 'each of a plurality of A' may refer to each of all components included in a plurality of A, or may refer to each of some components included in a plurality of A.

Before describing various embodiments of the present disclosure, the terms used will be explained.

In embodiments of the present disclosure, 'document image' includes images generated by photographing or scanning printed documents and images related to electronic documents (e.g., electronic receipts, electronic business cards, electronic identification cards, etc.) generated by programs. can do. A document image may contain at least one character.

In embodiments of the present disclosure, 'target text' may be related to a word/sentence/phrase that must be extracted from a plurality of characters included in a document image. For example, the target text may be a string associated with an entity value.

In embodiments of the present disclosure, 'entity' may be a string that becomes a key for recognizing target text. A plurality of characters may be recorded in a document, and among the plurality of characters, a string that serves as a standard for recognizing the target text may be an entity. For example, if the first document is a business registration certificate, the first entity may be the corporation name, the second entity may be the representative, and the third entity may be the business location, and the first target text may be the first text written in connection with the corporation name, the second entity The target text may be a second text written in connection with the representative, and the third target text may be a third text written in the document in connection with the location of the business. Taking a database table as an example, the entity may be a field name (column name) included in the database table, and the target text may be a value/detailed data related to the field name.

Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.

Figure 1 is a schematic diagram illustrating a process for selecting a model set to be distributed, according to an embodiment of the present disclosure. As illustrated in FIG. 1 , multiple model sets may be created during model set creation process 110 . Here, each of the plurality of model sets may be different in at least one of a data set used for learning, a learning method, a learning period, a character set, or an entity set. Additionally, during the distribution target selection process 120, a target model set to be distributed to the customer may be determined from among a plurality of model sets. The customer may be a subscriber to a text recognition service. In Figure 1, it is illustrated that the target model set is selected as model set_2.

According to one embodiment, computing resources are allocated to the distributed model set, so that the model set can operate as a logically/physically independent object. Each of the plurality of model sets may include at least one machine learning model. According to one embodiment, the model set may include at least one of a detection model, a recognition model, or a parsing model. The detection model may be a machine learning model configured to detect a plurality of sub-areas in which characters are expressed from a document image. Additionally, the recognition model may be a machine learning model configured to recognize a plurality of characters from each of a plurality of sub-areas. In addition, the parsing model may be a machine learning model configured to extract a plurality of target texts by parsing a plurality of characters output from the recognition model, and output recognition result data including at least one extracted target text. there is.

According to one embodiment, the parsing model may extract a plurality of target texts by parsing a plurality of input characters based on an entity set. An entity set may contain multiple entities. The parsing model can extract a string associated with each of the plurality of entities included in the entity set as the target text from among the plurality of input characters. For example, when the entity set includes an entity called 'business registration number', the parsing model can extract a string representing detailed data about the business registration number from a plurality of input characters as the target text.

According to one embodiment, computing resources allocated to the model set may be used to analyze document images to recognize characters. With reference to FIGS. 4 and 5 , a method for pre-training a machine learning model included in a model set will be described.

Figure 2 is a schematic diagram showing a configuration 200 connected to enable communication between an information processing system 230 and a plurality of communication terminals 210_1, 210_2, and 210_3 according to an embodiment of the present disclosure. As shown in Figure 2, a plurality of communication terminals (210_1, 210_2, 210_3) may communicate with the information processing system 230 through the network 220. Additionally, the information processing system 230 may be a system for providing a total text recognition service. According to one embodiment, the information processing system 230 may include a large amount of computing resources, and may allocate some of the total computing resources as resources for operating a model set. According to some embodiments, the information processing system 230 may transmit the model set to a computing system (not shown) of a customer company that has subscribed to the text recognition total service. Here, the customer's computing system may include servers, storage devices, etc. included in the customer's computer room. Here, transmitting the model set to the customer's computing system may mean transmitting software (eg, programming code, data set, entity set, etc.) for operating the model set to the customer's computing system.

The network 220 may be configured to enable communication between a plurality of communication terminals 210_1, 210_2, and 210_3 and the information processing system 230. Depending on the installation environment, the network 220 may be, for example, a wired network 220 such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device, and RS-serial communication, a mobile communication network, or a WLAN (Wireless Network). It may be comprised of a wireless network 220 such as LAN), Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and may include communication methods utilizing communication networks that the network 220 may include (e.g., mobile communication networks, wired Internet, wireless Internet, broadcasting networks, satellite networks, etc.), as well as communication terminals (210_1, 210_2, 210_3), short-range wireless communication may also be included. For example, the network 220 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , may include one or more arbitrary networks such as the Internet. Additionally, the network 220 may include any one or more of network topologies including a bus network, star network, ring network, mesh network, star-bus network, tree or hierarchical network, etc. Not limited.

In one embodiment, information processing system 230 is one or more server devices and/or databases capable of storing, serving, and executing computer-executable programs (e.g., downloadable applications) and data, or cloud computing service-based It may include one or more distributed computing devices and/or distributed databases. The information processing system 230 may perform pre-training for each of a plurality of machine learning models. Here, each of the plurality of machine learning models may be a model learned based on the document type. For example, the first machine learning model may be a model learned to recognize characters from a first type of document image, and the second machine learning model may be a model learned to recognize characters from a second type of document image.

According to one embodiment, the information processing system 230 may store a data set for training each of a plurality of machine learning models. Additionally, the information processing system 230 may store a character set applied to each of a plurality of machine learning models. Additionally, the information processing system 230 may include an entity set that applies to each of a plurality of machine learning models.

Machine learning models, data sets, character sets, and entity sets can each be associated with a document type. For example, a first data set, a first machine learning model, a first set of characters, and a first set of entities may be associated with a first type of document, and a second data set, a second machine learning model, and a second character The set and the second entity set may be associated with a second type of document.

The information processing system 230 can provide a total text recognition solution. According to one embodiment, the information processing system 230 may generate a plurality of model sets and perform dictionary learning for each of the plurality of model sets. Additionally, the information processing system 230 may determine a target model set to be distributed to customers from among a plurality of model sets, and distribute the determined target model set to the target system. Here, the target system may be a computing system configured including computing resources included in the information processing system 230. As another example, the target system may be a computing system included in the customer's computer room. In one embodiment, the target system may recognize a plurality of characters and/or target text from document images received from the communication terminals 210_1, 210_2, and 210_3 using the distributed target model set.

According to one embodiment, the information processing system 230 may perform a test on each of a plurality of model sets and then determine a target model set based on the test results. Here, the test may be a test on a model set performed based on a plurality of document images. Additionally, a plurality of document images may include images received from a communication terminal and document images being stored in an information processing system. According to some embodiments, testing may be conducted with multiple document images equally applied to each of multiple model sets.

According to one embodiment, based on test results, evaluation data for each of a plurality of model sets may be obtained. The evaluation data may include at least one of the average delay time of the model set, the number of errors, the recognition accuracy of the model set, or the confidence score of the model set. Here, the delay time may be the time required from the document image being input to the character recognition result (eg, parsing data) being output. The information processing system can measure the time from when a document image is input to the model set until recognition result data is output from the model set as delay time.

Recognition accuracy may be a value that quantifies the degree of agreement between the target text recognized in the model set and the correct answer data. According to one embodiment, recognition accuracy may be measured by applying a labeled test data set to a model set. Here, the labeled test data set may include a plurality of test data associated with the correct answer data. A specific example of calculating recognition accuracy for a model set will be described later with reference to FIG. 6.

The confidence score may be a probability associated with accuracy. In other words, the confidence score can be related to the predicted value, and the recognition accuracy can be related to the match rate with the standard correct answer data. A specific example of how the trust score is calculated will be described later with reference to FIG. 10.

Based on the evaluation data of each of the plurality of model sets, performance indicators for each of the plurality of model sets may be calculated and output, and based on the calculated plurality of performance indicators, a target model set among the plurality of model sets may be selected. can be decided. A performance indicator for the model set may be calculated based on at least one of a trust score for the model set included in the evaluation data, recognition accuracy of the model set, number of errors, resource group status, or delay time. For example, the information processing system may calculate the average value of delay time, number of errors, and average trust score for each target text based on a plurality of state data for a specific model set collected during testing. Here, the state data is data collected while the model set is operating and may include delay time, whether an error occurred, and a trust score for each target text.

Additionally, the information processing system can calculate a trust score for a specific model set by weighting and summing the average trust score for each target text. At this time, the weight may be applied equally to each target text, or a higher weight may be applied to important target texts. According to one embodiment, the information processing system may generate evaluation data including average values of latency, number of errors, and confidence scores for a set of models. In addition, the information processing system may calculate a performance indicator for a specific model set based on at least one of the average value of the delay time included in the evaluation data, the number of errors, the confidence score for the specific model set, or the recognition accuracy of the model set. You can. For example, the information processing system applies a first weight to the average value of the delay time, a second weight to the number of errors, a third weight to the confidence score, and a fourth weight to the recognition accuracy of the model set, By summing the average weighted delay time, number of errors, confidence score, and recognition accuracy, performance metrics for a specific model set can be calculated. Weights may be applied so that the size of the performance indicator is inversely proportional to the average value of delay time, proportional to the confidence score, proportional to recognition accuracy, and inversely proportional to the number of errors. The information processing system 230 may determine the model set with the highest performance index as the target model set to be distributed.

According to some embodiments, the information processing system 230 may determine a target model set from a plurality of model sets based on the document type. For example, the information processing system 230 may receive a document type to be analyzed from a customer and determine a model set associated with the received document type as the target model set. For example, when a receipt is received as a document type, the information processing system may determine the model set associated with the receipt as the target model set.

According to some embodiments, the information processing system 230 may determine a target model set from a plurality of model sets based on a user input for determining the target model set. For example, a customer may transmit a user input to the information processing system 230 to determine a model he needs among a plurality of model sets as a target model set.

According to one embodiment, when distributing the target model set, the information processing system 230 may transmit at least one of a character set, an entity set, or a data set associated with the target model set to the target system. Here, the character set may include at least one of a language to be recognized or a plurality of special characters. Additionally, an entity set may include multiple entities. Additionally, the information processing system 230 may determine computing resources allocated to the target model set. In this case, the target system may operate at least one machine learning model included in the target model set using the determined computer resources.

According to one embodiment, the information processing system 230 may monitor the operating state of at least one of a distributed target model set or a newly created model set. For example, when the target model set is operated, the target system transmits status data for each of at least one model set included in the target model set to the information processing system 230, and the information processing system 230 receives the status data. Based on the data, the operating status of the target model set can be monitored.

Communication terminals 210_1, 210_2, and 210_3 may capture, receive, and/or collect document images and transmit them to the information processing system 230. In Figure 2, a mobile phone or smartphone 210_1, a tablet computer 210_2, and a laptop or desktop computer 210_3 are shown as examples of terminals that transmit document images to the information processing system 230, but are not limited thereto and are communication terminals. (210_1, 210_2, 210_3) may be any computing device capable of wired and/or wireless communication and capable of running a web browser, mobile browser application, or character recognition application. In addition, in Figure 2, three communication terminals (210_1, 210_2, 210_3) are shown as communicating with the information processing system 230 through the network 220, but this is not limited to this, and a different number of communication terminals are connected to the network ( It may be configured to communicate with the information processing system 230 through 220).

In some embodiments, the communication terminals 210_1, 210_2, and 210_3 provide a document image to the information processing system 230 and then retrieve a plurality of characters and/or target text recognized from the document image from the information processing system 230. You can receive it.

In some embodiments, the communication terminals 210_1, 210_2, and 210_3 may be point of sale (POS) terminals installed in a store. When payment for a product is completed, the communication terminals 210_1, 210_2, and 210_3 may generate a receipt image and transmit it to the information processing system 230.

According to one embodiment, a user interface may be used to obtain various information about the model set. Customers can use the user interface to obtain information about the model set and check the status of the model set. Various examples of user interfaces associated with model sets will be described later with reference to FIGS. 6 to 13. This user interface may be provided through the information processing system 230 and/or the target system.

Information processing system 230 may include a database. A database can store multiple state data. Here, the state data may be data associated with the model set and state. Additionally, the database may store any log data associated with the operation of the model set. For example, log data may include, but is not limited to, request reception time, response message transmission time, delay time, etc.

The status of at least one of the deployed model set or the new model set may be monitored for a predetermined period of time. Based on the status of the monitored model set, it may be determined whether the deployed model set/new model set will ultimately be used in the target system. The model set decided to be used in the target system replaces the model set already in use, and the document image is applied to the replaced model set, so that recognition of characters included in the document image can be performed.

Figure 3 is a block diagram showing the internal configuration of a communication terminal 210 and an information processing system 230 according to an embodiment of the present disclosure. The communication terminal 210 may refer to any computing device capable of wired/wireless communication, for example, a mobile phone or smartphone 210_1, a tablet computer 210_2, and a laptop or desktop computer 210_3 in FIG. 2. It may include etc. As shown, the communication terminal 210 may include a memory 312, a processor 314, a communication module 316, and an input/output interface 318. Similarly, information processing system 230 may include memory 332, processor 334, communication module 336, and input/output interface 338. As shown in FIG. 3, the communication terminal 210 and the information processing system 230 are configured to communicate information and/or data through the network 220 using respective communication modules 316 and 336. It can be. Additionally, the input/output device 320 may be configured to input information and/or data to the communication terminal 210 through the input/output interface 318 or to output information and/or data generated from the communication terminal 210.

Memories 312 and 332 may include any non-transitory computer-readable recording medium. According to one embodiment, the memories 312 and 332 are non-permanent mass storage devices such as read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. It can be included. As another example, non-perishable mass storage devices such as ROM, SSD, flash memory, disk drive, etc. may be included in the communication terminal 210 or the information processing system 230 as a separate persistent storage device that is distinct from memory. Additionally, an operating system and at least one program code (for example, a code for recognizing characters from a document image, etc.) may be stored in the memories 312 and 332.

These software components may be loaded from a computer-readable recording medium separate from the memories 312 and 332. This separate computer-readable recording medium may include a recording medium directly connectable to the communication terminal 210 and the information processing system 230, for example, a floppy drive, disk, tape, DVD/CD- It may include computer-readable recording media such as ROM drives and memory cards. As another example, software components may be loaded into the memories 312 and 332 through a communication module rather than a computer-readable recording medium. For example, at least one program is based on a computer program (e.g., a program for character recognition) that is installed by files provided over the network 220 by developers or a file distribution system that distributes installation files for applications. Thus, it can be loaded into the memories 312 and 332.

The processors 314 and 334 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processors 314 and 334 by memories 312 and 332 or communication modules 316 and 336. For example, processors 314 and 334 may be configured to execute received instructions according to program codes stored in recording devices such as memories 312 and 332.

The communication modules 316 and 336 may provide a configuration or function for the communication terminal 210 and the information processing system 230 to communicate with each other through the network 220, and the communication terminal 210 and/or information processing The system 230 may provide a configuration or function for communicating with other communication terminals or other systems (for example, a separate cloud system, etc.). For example, a request (e.g., a character recognition request, etc.) generated by the processor 314 of the communication terminal 210 according to a program code stored in a recording device such as the memory 312 is controlled by the communication module 316. Accordingly, it may be transmitted to the information processing system 230 through the network 220. Conversely, the control signal or command provided under the control of the processor 334 of the information processing system 230 is transmitted through the communication module 316 of the communication terminal 210 through the communication module 336 and the network 220. It can be received by the communication terminal 210.

The input/output interface 318 may be a means for interfacing with the input/output device 320. As an example, an input device may include devices such as a keyboard, a microphone, a mouse, and a camera including an image sensor, and an output device may include devices such as a display, a speaker, a haptic feedback device, etc. As another example, the input/output interface 318 may be a means for interfacing with a device that has a structure or function for performing input and output, such as a touch screen, integrated into one. For example, the processor 314 of the communication terminal 210 uses information and/or data provided by the information processing system 230 or another communication terminal when processing instructions of a computer program loaded in the memory 312. The service screen configured as follows may be displayed on the display through the input/output interface 318.

In FIG. 3 , the input/output device 320 is shown not to be included in the communication terminal 210, but the present invention is not limited to this and may be configured as a single device with the communication terminal 210. Additionally, the input/output interface 338 of the information processing system 230 is connected to the information processing system 230 or means for interfacing with a device (not shown) for input or output that the information processing system 230 may include. It can be. In FIG. 3 , the input/output interfaces 318 and 338 are shown as elements configured separately from the processors 314 and 334, but the present invention is not limited thereto, and the input/output interfaces 318 and 338 may be configured to be included in the processors 314 and 334. there is.

Communication terminal 210 and information processing system 230 may include more components than those in FIG. 3 . However, there is no need to clearly show most prior art components. According to one embodiment, the communication terminal 210 may be implemented to include at least some of the input/output devices 320 described above. Additionally, the communication terminal 210 may further include other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database.

The processor 334 of the information processing system 230 may be configured to manage, process, and/or store information and/or data received from a plurality of communication terminals, including the communication terminal 210, and/or a plurality of external systems. there is. Information and/or data processed by the processor 334 may be provided to the communication terminal 210 through the communication module 336. In one embodiment, processor 334 may distribute resources associated with the model set to customers via communications module 336. Resources associated with the deployed model set may include hardware resources and/or software resources, which may be used to retrieve document images from the communication terminal 210, memory 332, and/or external storage device. Received, a plurality of characters can be recognized from the document image.

For example, the information processing system 230 may include a cloud computing system, allocate some hardware included in the cloud computing system for a distributed model set, and then transfer software resources associated with the model set to the allocated hardware. there is. As another example, if the customer owns a separate computing system, information processing system 230 may transfer resources associated with the model set to the customer's computing system. In this case, a character recognition service may be provided through the customer's computing system.

FIG. 4 is an example diagram illustrating how the machine learning models 412, 414, and 416 included in the model set 410 are learned, according to an embodiment of the present disclosure. As illustrated in FIG. 4, a data set including a plurality of training images 420_1 to 420_n and the correct answer data 440 is used to create a detection model 412 and a recognition model 414 included in the model set 410. and parsing model 416 can be learned. Here, the detection model 412, recognition model 414, and parsing model 416 may include a machine learning model (eg, artificial neural network, etc.). In Figure 4, the detection model 412, recognition model 414, and parsing model 416 are illustrated as being separated, but two or more models may be integrated and implemented as one machine learning model.

According to one embodiment, the plurality of training images 420_1 to 420_n included in the data set may be document images associated with a specific document type. That is, the plurality of training images 420_1 to 420_n included in the data set may be the same type of training image.

The detection model 412 included in the model set 410 may be a machine learning model configured to detect a plurality of sub-areas in which characters are expressed from the training image. Additionally, the recognition model 414 included in the model set 410 may be a machine learning model configured to recognize a plurality of characters from each of a plurality of sub-areas.

According to one embodiment, the detection model 412 may receive a training image, detect a plurality of sub-areas in which characters are expressed in the entire area of the learning image, and then output the plurality of detected sub-areas. Here, the sub-area is an area corresponding to a portion of the entire area of the learning image and may be an area where characters are displayed. That is, blank spaces in which no characters are displayed may not be included in the sub-area.

Additionally, the recognition model 414 may receive a plurality of sub-areas output from the detection model 412, recognize a plurality of characters included in each of the plurality of sub-areas, and then output the recognized plurality of characters. . According to one embodiment, the recognition model 414 may recognize a plurality of characters included in each of a plurality of sub-areas based on the target language included in the character set. Here, the character set may include a language and special characters to be recognized, and different character sets for each document type may be stored in the information processing system. The target language may be a language that is subject to recognition. For example, if the character set includes Hangul and does not include English, characters related to English may be excluded from recognition, and characters related to Hangul may be recognized.

According to one embodiment, the recognition model 414 may recognize special characters included in each of a plurality of sub-areas based on a list of special characters included in the character set. That is, if a special character included in the special character list is included in the sub-area, the recognition model 414 can recognize the special character. If a special character that is not included in the special character list is included in the sub-area, the recognition model 414 may not recognize the special character.

The parsing model 416 extracts a plurality of target texts by parsing a plurality of characters output from the recognition model 414, and outputs parsing result data 430 including at least one extracted target text. can do. The parsing result data 430 may further include a plurality of sub-areas that are a result of detection by the detection model 412 and/or a plurality of characters that are a result of recognition by the recognition model 414. According to one embodiment, the parsing model 416 may extract a plurality of target texts by parsing a plurality of input characters based on the entity set. At this time, the parsing model 416 may extract a string associated with each of the plurality of entities included in the entity set as the target text from among the plurality of input characters.

A loss value between the parsing result data 430 and the correct answer data 440 is calculated, and the calculated loss value is reflected in the parsing model 416 so that the weight of at least one node constituting the parsing model 416 can be adjusted. there is. For example, the match rate between the plurality of target texts included in the parsing result data 430 and the plurality of target texts included in the correct answer data 440 may be calculated as a loss value and fed back to the recognition model 414. . As another example, various functions can be used to calculate loss values. The correct answer data 440 is the correct answer data 440 associated with the training image input to the detection model 412, and may include a plurality of target texts that must be parsed. The correct answer data 440 may be data generated through a labeling operation.

Additionally, the correct answer data 440 may further include a list of characters. In this case, a loss value is calculated between a character set including a plurality of characters output from the recognition model 414 and the character list included in the correct answer data 440, and the calculated loss is sent to the recognition model 414. Through feedback, the weight of at least one node constituting the recognition model 414 may be adjusted. Here, the character set may be a list of characters. For example, the match rate between a character set including a plurality of characters output from the recognition model 414 and the character list included in the correct answer data 440 may be calculated as a loss value and reflected in the recognition model 414. As another example, various functions can be used to calculate loss values.

Additionally, the correct answer data 440 may further include a pixel range for each of the plurality of sub-areas. For example, the correct answer data 440 may include the pixel range of each sub-area where a character is located in the training image. In this case, the loss value between the pixel range for each sub-region included in the correct answer data 440 and the pixel range for each sub-region output from the detection model 412 is calculated, and the calculated loss value is reflected in the detection model 412. , the weight of at least one node constituting the detection model 412 may be adjusted. For example, the matching rate between the pixel range for each sub-region included in the correct answer data 440 and the pixel range for each sub-region output from the detection model 412 may be calculated as a loss value and reflected in the recognition model 414. As another example, various functions can be used to calculate loss values.

When a predetermined condition is met, learning is stopped, and the weight of at least one node constituting each of the detection model 412, recognition model 414, and parsing model 416 may be determined. Here, the predetermined condition may be a state in which the number of learning times reaches a threshold number or a loss value is less than a threshold value. A model set including the detection model 412, recognition model 414, and parsing model 416 with weights determined in this way may be stored in the information processing system.

Figure 5 is an exemplary diagram showing an artificial neural network model 500 according to an embodiment of the present disclosure. The artificial neural network model 500 is an example of a machine learning model, and in machine learning technology and cognitive science, it may be a statistical learning algorithm implemented based on the structure of a biological neural network or a structure for executing the algorithm. there is.

According to one embodiment, the artificial neural network model 500 has nodes, which are artificial neurons that form a network through the combination of synapses, as in a biological neural network, repeatedly adjusting the weights of synapses to determine the correct input corresponding to a specific input. By learning to reduce the error between the output and the inferred output, a machine learning model with problem-solving capabilities can be represented. For example, the artificial neural network model 500 may include a random probability model, a neural network model, etc. used in artificial intelligence learning methods such as machine learning and deep learning. According to one embodiment, the artificial neural network model 500 may be associated with at least one of a detection model, a recognition model, or a parsing model illustrated in FIG. 4 .

The artificial neural network model 500 can be implemented as a multilayer perceptron (MLP) consisting of multiple layers of nodes and connections between them. The artificial neural network model 500 according to this embodiment can be implemented using one of various artificial neural network model structures including MLP. As shown in FIG. 5, the artificial neural network model 500 includes an input layer 520 that receives an input signal or data 510 from the outside, and an output layer that outputs an output signal or data 550 corresponding to the input data. (540), located between the input layer 520 and the output layer 540, receives n signals from the input layer 520, extracts the characteristics, and transmits them to the output layer 540 (where n is a positive integer) It may be composed of hidden layers 530_1 to 530_n. Here, the output layer 540 may receive signals from the hidden layers 530_1 to 530_n and output data to the outside.

The learning method of the artificial neural network model 500 includes a supervised learning method that learns to optimize problem solving by inputting a teacher signal (correct answer), and an unsupervised learning method that does not require a teacher signal. ) There is a way. In one embodiment, the artificial neural network model 500 may be supervised to recognize characters from document images. Additionally or alternatively, the artificial neural network model 500 may be supervised to recognize target text from document images.

In this way, a plurality of input data and a plurality of output data corresponding to the input layer 520 and the output layer 540 of the artificial neural network model 500 are matched, respectively, and the input layer 520, the hidden layers 530_1 to 530_n, and By adjusting the synapse values between nodes included in the output layer 540, learning can be performed so that the correct output corresponding to a specific input can be extracted. Through this learning process, characteristics hidden in the input data of the artificial neural network model 500 can be identified, and the nodes of the artificial neural network model 500 can be configured to reduce the error between the output data calculated based on the input data and the target output. Synapse values (or weights) between them can be adjusted. The artificial neural network model 500 learned in this way may output a plurality of characters and/or a plurality of target texts from the document image in response to the input document image.

6 is a diagram illustrating a user interface 600 associated with learning a model set, according to one embodiment of the present disclosure. As illustrated in FIG. 6 , information about a plurality of model sets may be output to the first area 610. Multiple model sets may be generated by the information processing system. The model set illustrated in Figure 6 can be configured to recognize characters from receipt images.

Each of the plurality of model sets generated by the information processing system may differ in at least one of a data set used for learning, a learning method, a learning period, a character set, or an entity set. For example, even if the same data set is used to learn the first model set and the second model set, if at least one of the learning method, learning period, character set, or entity set is different, the first model set and the second model set are different from each other. Can be managed as a set of different models. As another example, when the data set used for training the first model set includes additional data, a third model set is created that is separate from the first model set, and a third model associated with the data set that includes the additional data. Sets can be managed separately.

In the first area 610 of FIG. 6, a list of model sets with identifiers 'Modelset-0001', 'Modelset-0002-Test', and 'Modelset-0003' is displayed. Additionally, recognition accuracy for each model set may be output in the first area 610 of FIG. 6 .

According to one embodiment, the information processing system may calculate and output recognition accuracy for each of a plurality of target texts associated with a plurality of entities based on recognition result data and correct answer data output from a model set. In Figure 6, the recognition accuracy of target text associated with each of 'store name', 'store business registration number', 'store address', 'store transaction date', 'menu product name', and 'single product name' is measured in the first area (610). ) can be output. According to one embodiment, a data set associated with learning may include correct answer data associated with a document image, and the information processing system calculates recognition accuracy, which represents the degree of agreement between the target text recognized from the model set and the target text included in the correct answer data. can do. For example, if the first target text recognized from the first document image is 'baker' and the first correct answer data associated with the first target text is 'bakery', the recognition accuracy for the first target text is '0.75'. can be calculated. Recognition accuracy for each target text can be updated as learning progresses repeatedly. For example, the recognition accuracy for the target text output in the first area 610 of FIG. 6 is an average value for recognition accuracy, and may be updated while learning is in progress. As another example, the recognition accuracy of the target text output to the first area 610 of FIG. 6 may be the recognition accuracy for a recently recognized target text.

Additionally, recognition accuracy for the model set may be calculated based on recognition accuracy for each of a plurality of target texts. For example, an average value for the recognition accuracy of a plurality of target texts may be calculated as the recognition accuracy for the model set. As another example, the recognition accuracy for each of the plurality of target texts is weighted and summed, and the value divided by a predetermined value may be calculated as the recognition accuracy for the model set. At this time, based on the importance of the target text, a relatively large weight may be applied to the recognition accuracy of the target text with high importance.

Referring to the model set with the identifier 'Modelset-0001' output in the first area 610 of FIG. 6, the recognition accuracy for the first target text associated with 'store name' is '0.824', and 'store business registration' It is exemplified that the recognition accuracy for the second target text associated with 'number' is '0.953', and the recognition accuracy for the third target text associated with 'store address' is '0.797'. Additionally, in Figure 6, the recognition accuracy for the model set with the identifier 'Modelset-0001' is illustrated as '0.894'.

As illustrated in the second area 620 and the third area 630 of FIG. 6, the learning status for each of the plurality of model sets may be output. In the second area 620 of FIG. 6, information about the model set being learned is illustrated as being output. Additionally, information on a model set whose learning has been stopped is shown as being output in the third area 630 of FIG. 6 . The identifier for each model set and the identifier of at least one machine learning model included in the model set may be output in each of the second area 620 and the third area 630. Additionally, the learning time may be output to each of the second area 620 and the third area 630.

At least one of the plurality of model sets may be determined as the target model set to be distributed to the customer's target system. According to one embodiment, the information processing system may perform a test on each of a plurality of model sets and, based on the performed test, determine a target model set to be distributed to the target system. In FIG. 6, the model set (Modelset-0001) displayed with the rocket-shaped icon 612 is illustrated as a model set with a distribution history. In some embodiments, a separate icon indicating the target model set may be displayed around the identifier of the model set determined to be distributed.

FIG. 7 is a diagram illustrating a user interface 700 including detailed information about a model set, according to an embodiment of the present disclosure. When a user input requesting a detailed view of a specific model set is received, a user interface 700 including detailed information as shown in FIG. 7 may be output. In Figure 7, detailed information about a model set with the identifier "Medelset-0002" is illustrated as being output.

As illustrated in the first area 710 of FIG. 7, information about a plurality of machine learning models included in the model set may be output. In the first area 710 of FIG. 7, information on a detection model with the identifier “Detector-0001”, a recognition model with the identifier “Recognizer-0001”, and a parsing model with the identifier “Parser-0001” are displayed. It is done. Additionally, information about the data set used for training each machine learning model may be output to the first area 710. In the first area 710 of FIG. 7, the data set applied to the machine learning model is illustrated as 'Dateset-001'.

Additionally, as illustrated in the second area 720 of FIG. 7, general information about the model set may be output. General information about the model set may include the speed of the model set, GPU allocation information, API endpoint information, allocated memory information, production history information, creation date, etc. Here, the API endpoint information may be information related to the target system where the model set is distributed.

As illustrated in the third area 730 of FIG. 7, configuration information of the character set and entity set may be output. That is, information about the character set applied to the model set and information about the entity set may be output in the third area 730.

As illustrated in the fourth area 740 of FIG. 7, the recognition accuracy of each model set may be output. In the fourth area 740 of FIG. 7, 'F1 score' is illustrated as recognition accuracy.

The model set currently being output (e.g., the model set associated with the Modelset-0002 identifier) may be distributed. For example, when a user input for selecting the “Deply to Endpoint” menu in the upper right of FIG. 7 is received, the model set can be distributed to the target system.

8 is a diagram illustrating a user interface 800 for distributing a model set, according to one embodiment of the present disclosure. Referring to FIG. 8, when a model set is distributed to a target system, computing resources allocated to the model set may be determined. Here, the computing resource may be a resource for a computing device that receives the deployed model set. For example, when a deployed model set is transferred to a customer's computing system, computing resources included in the customer's computing system may be allocated for the model set. As another example, the information processing system may include a cloud computing system, and when the deployed model set is transmitted to the cloud computing system, computing resources included in the cloud computing system may be allocated for the model set.

Through the user interface 800 as shown in FIG. 8, the distributed model set and computing resources can be determined. Additionally, the rate of input data can be determined and the monitoring period can be determined.

The identifier of the model set to be distributed can be input into the first area 810 of FIG. 8, and the computing resources allocated to each of the plurality of machine learning models constituting the model set can be input into the second area 820. It can be. Additionally, computing resources allocated to the model set may be determined and output in the third area 830 of FIG. 8 . According to one embodiment, based on the number of computing resources (eg, workers) input to the second area 820, computing resources allocated to the model set may be determined and output to the third area 830. The amount of computing resources allocated to the model set may be less than or equal to the amount of computing resources allocated to a plurality of machine learning models included in the model set.

In FIG. 8 , the number of GPUs (Graphics Processing Units) is illustrated as being input to the second area 820 and the third area 830, but the present disclosure is not limited to this, and the second area 820 and the third area 830 Memory capacity, network traffic, disk capacity, etc. may be additionally entered into area 830.

The ratio of input data may be input into the fourth area 840 of FIG. 8 . Here, the ratio of input data may be related to the ratio of input data applied to the model set (Current Modelset) in use in the target system and the ratio of input data applied to the distributed model set (New Modelset). If the proportion of input data applied to the model set currently in use in the target system is 70% and the proportion of input data applied to the deployed model set is 30%, 70% of the input data collected during the monitoring period is currently used. 30% of the input data collected during the monitoring period can be applied to the deployed model set. Here, the input data may include a document image.

The monitoring period for the distributed model set can be entered in the fourth area 850 of FIG. 8.

9 is a diagram illustrating a user interface 900 for monitoring the status of a new model set, according to one embodiment of the present disclosure. When user input for monitoring a new model set is received, user interface 900 may be output. Here, the new model set may be a model set scheduled for distribution. The user interface 900 illustrated in FIG. 9 can be used to evaluate the stability and performance of a new model set.

As illustrated in the first area 910 of FIG. 9, the monitoring period of the new model set may be output. Additionally, as illustrated in the second area 920 of FIG. 9, the status of the new model set may be monitored and output. In the second area 920 of FIG. 9, the resource group status allocated to a plurality of machine learning models included in the new model set is illustrated as being output. Here, the resource group may include the number of processors, memory capacity, disk capacity, network traffic, etc. allocated by the machine learning model, and the resource group status includes at least one of processor resources, memory resources, disk resources, network resources, etc. This can be a status for a resource group. In the second area 920 of FIG. 9, the number of processors is shown as a resource group. That is, referring to the second area 920 of FIG. 9, the model set with the identifier 'Modelset-0004' includes a detection model, a recognition model, and a parsing model, 6 processors are assigned to the detection model, and the recognition model In the example, 12 processors are allocated, and 4 processors are allocated to the parsing model.

In FIG. 9 , when the resource group status is the first color (eg, green), it may indicate that the related processor is smoothly processing work. In FIG. 9, when the resource group status is a second color (eg, red), it may indicate that the related processor is processing the task delayed, fails to process the task, or is waiting without performing the operation.

According to one embodiment, an information processing system may collect status data for a new model set. Here, the state data may include state data for each of a plurality of machine learning models included in the new model set. For example, if the new model set includes a detection model, a recognition model, and a parsing model, the state data for the new model set includes first state data for the detection model, second state data for the recognition model, and second state data for the parsing model. 3 Can contain state data. Each of the first state data, second state data, and third state data may include various states such as processor state, processor utilization, memory utilization, etc.

The information processing system may determine and output the state for the detection model based on the first state data, determine and output the state for the recognition model based on the second state data, and determine and output the state for the recognition model based on the third state data. Thus, the status of the parsing model can be determined and output.

The resource group status for the new model set may be determined based on the resource group status for a plurality of machine learning models included in the new model set. For example, if the new model set includes a detection model, a recognition model, and a parsing model, the information processing system may configure a first resource group state for the detection model, a second resource group state for the recognition model, and a third resource group state for the parsing model. The group status may be identified, and based on the identified first resource group status, second resource group status, and third resource group, the resource group status of the model set may be determined and output.

Monitoring history for a new model set may be output in the third area 930 of FIG. 9 .

A user interface 900 as shown in FIG. 9 may be provided to the customer, and the customer may check the status of the new model set through the user interface 900 and decide whether to use the new model set.

10 is a diagram illustrating a user interface 1000 containing trust scores of a deployed model set, according to one embodiment of the present disclosure. Figure 10 illustrates the confidence score for a set of models that are deployed and in use. Here, the confidence score may be a probability associated with accuracy. For example, after the probability of matching one target text and each entity obtained through image recognition is calculated, the probability with the highest probability among the calculated match probabilities may be determined and output as the trust score for the target text. For example, a first match probability associated with a first target text and a first entity, a second match probability associated with a second entity associated with the first target text, ??, the nth entity associated with the first target text (where, Among the nth matching probabilities associated with (n is a natural number of 3 or more), the highest matching probability may be determined and output as the trust score for the first target text. Calculation of these trust scores can be performed through an information processing system.

According to one embodiment, when a user input requesting to view the trust score for a specific model set is received, a user interface 1000 as shown in FIG. 10 may be output. As illustrated in FIG. 10 , based on the state data of the model set, a trust score 1010 for the model set may be calculated and output. According to some embodiments, the trust score 1010 may be calculated using various functions for calculating the trust score 1010.

According to one embodiment, a trust score for a model set may be calculated and output based on a plurality of trust scores for a plurality of data (eg, target text) output from the model set. For example, when a plurality of target texts are output from a model set, the average trust score of the target texts is calculated, and the calculated average trust score may be determined and output as the trust score 1010 of the model set. The state data may include a trust score for each target text, and a trust score 1010 for the model set may be calculated based on the trust score for each target text included in the state data.

The trust score 1010 output as a number in FIG. 10 may be the trust score of the most recently calculated model set. As another example, the trust score 1010 output as a number in FIG. 10 may be an average value of trust scores of model sets calculated for multiple periods.

According to one embodiment, based on state data of the model set collected according to a pre-designated time period, the trust score of the model set for each time period may be calculated and output in the form of a graph (1020). In Figure 10, it is illustrated that the trust score for the model set is calculated by date, and the calculated trust score by date is output as a line graph 1020.

FIG. 11 is a diagram illustrating a user interface 1100 containing error information for a model set, according to an embodiment of the present disclosure. In response to receiving a user input for viewing error information for a model set, a user interface 1100 as shown in FIG. 11 may be output. As illustrated in FIG. 11, error information 1110 for the model set may be output. Error information 1110 may include the number of input errors. Additionally or alternatively, the error information 1110 may include counts of various types of errors related to response errors, memory errors, processor errors, etc. where the model set does not respond even if the set exceeds a predetermined time. The number of errors for the model set may be calculated based on the number of input errors, number of response errors, number of memory errors, number of processor errors, etc.

The information processing system may measure the number of times a document of an unauthorized type is input into the model set (i.e., the number of input errors), and generate and output error information 1110 based on the measured number of input errors. For example, if the model set is trained to process document images of the first type (e.g., receipt images), the information processing system may select document images associated with types other than the first type (e.g., business registration certificate images, etc.) from the model set. The number of input errors can be increased by determining that the document image is not permitted. As another example, when a document image having a file format not supported (i.e., an unsupported file extension) is input to the model set, the number of input errors for documents of an unauthorized type may increase.

Error information 1110 output as a number in FIG. 11 may be generated based on the number of input errors or the number of various types of errors measured during a predetermined period. According to one embodiment, based on the number of errors in the model set measured according to a pre-designated time period, error information of the model set for each time period may be calculated and output in the form of a graph 1120. In FIG. 11, it is illustrated that the error information of the model set is calculated by date, and the calculated error information by date is output as a line graph 1120.

FIG. 12 is a diagram illustrating a user interface 1200 including resource group status for a model set, according to one embodiment of the present disclosure. In response to receiving a user input for viewing the resource group status for the model set, a user interface 1200 as shown in FIG. 12 may be output. As illustrated in Figure 12, resource group status 1210 for the model set may be output. Here, the resource group status 1210 may be a status for at least one computing resource allocated to the model set. For example, the resource group may include the number of processors allocated to the model set, memory capacity, disk capacity, network traffic, etc., and the resource group status 1210 may include at least processor resources, memory resources, disk resources, network resources, etc. The status may be for a resource group that contains one. In Figure 12, the resource group state is illustrated as 'worker state'.

The resource group status for the model set may be determined based on the resource group status for a plurality of machine learning models included in the model set. For example, if the model set includes a detection model, a recognition model, and a parsing model, the information processing system may configure a first resource group state for the detection model, a second resource group state for the recognition model, and a third resource group state for the parsing model. The status may be identified, and based on the identified first resource group status, second resource group status, and third resource group, the resource group status of the model set may be determined and output. In Figure 12, when the resource group status is the first color (eg, green), it may indicate that the related model set is processing work smoothly. In FIG. 12, when the resource group status is a second color (eg, red), it may indicate that the related model set is processing the task with delay, failing to process the task, or waiting without performing the operation.

As illustrated in FIG. 12, the first resource group status for the detection model, the second resource group status for the recognition model, and the third resource group status for the parsing model included in the model set are output to the specific area 1220. It can be. In the second area 1220 of FIG. 12, it is illustrated that processor-related information is output as information about the resource group status. In Figure 12, two processors are assigned to a detection model with the identifier 'Detector-0001', and one of the two processors is processing the task smoothly, but one processor is waiting for the task or has failed to process the task. It is done.

FIG. 13 is a diagram illustrating a user interface 1300 associated with a recognition result, according to an embodiment of the present disclosure. When a user input for viewing the document recognition result is received, a user interface 1300 as shown in FIG. 13 may be output.

An image of a document that is to be recognized may be output in the first area 1310 of the user interface 1300. Additionally, the recognition result may be output to the second area 1320 of the user interface 1300. The parsing result is illustrated as being output in the second area 1320 of FIG. 13 .

The document image output to the first area 1310 may include a plurality of squares. Here, the rectangle may be a sub-area where characters are displayed. That is, the rectangular area may be a sub-area recognized through the detection model.

The second area 1320 may include a parsing result (i.e., recognition result) including the target text recognized based on the entity set. According to one embodiment, the parsing result output from the parsing model may be output to the second area 1320. As illustrated in the second area 1320, a list of entities may be displayed in the leftmost column, target text associated with the entity may be displayed in the middle column, and a confidence score may be displayed in the rightmost column.

According to one embodiment, a confidence score for target text recognition may be output together with the parsing result from a parsing model that outputs the parsing result. In contrast, various functions for calculating the trust score can be used to calculate the trust score for recognition of the target text.

The user can compare the document image output in the first area 1310 and the second area 1320 to verify whether the parsing result is accurate. According to one embodiment, when a specific target text is selected for editing among a plurality of target texts output to the second area 1320, at least one character associated with the selected specific target text is visualized through a specific graphic element It may be output in area 1 1310. In the second area 1320, when the target text 1322 located in the third line from the top is selected, a plurality of characters associated with the selected target text 1322, that is, the registration number 1312, are highlighted and visualized in the document image. It is shown that this has been done. According to this visualization effect, users can easily verify whether the target text has been parsed accurately by comparing characters associated with the target text one-to-one.

If an incorrect target text is found among the target texts included in the parsing result, the user can modify the parsing result using the user interface 1300. For example, if the 'registration number' displayed in the document image in the second area 1320 is recognized as 'registration number', the user may select 'registration number' in the second area 1320 containing the parsing result. ‘Registration number’ can be modified to ‘Registration number’. In the second area 1320 of FIG. 13, the target text 1322 located third from the top is illustrated as being changed to 'registration number'.

If the save menu 1324 is entered after the target text 1322 is changed, the parsing result (i.e., recognition result) may be changed. That is, the parsing result may be changed to include the target text 1322 changed by the user. In this case, additional learning data may be generated based on the changed parsing results. That is, learning data including a learning image displayed in the first area 1310 and a changed parsing result may be generated. Here, the changed parsing result can be used as ground truth data. The generated training data can be used to further train at least one machine learning model included in the model set. Specifically, the information processing system may additionally learn at least one machine learning model included in the model set using the generated training data. That is, the user can conveniently verify the recognition result through the user interface 1300 as shown in FIG. 13. If the recognition result is modified, training data including the corrected recognition result as correct answer data is generated, and then the generated training data can be used for additional training of the model set.

According to one embodiment, a plurality of characters and/or target text recognized using the model set may be stored in a database included in or managed by the information processing system. Characters and/or target texts stored in the database can be used in various fields such as subscriber management, consumption tendency pattern analysis, etc. in the future.

FIG. 14 is a flowchart illustrating a method 1400 for distributing a model set, according to an embodiment of the present disclosure. The method 1400 shown in FIG. 14 is only an example to achieve the purpose of the present disclosure, and of course, some steps may be added or deleted as needed. Additionally, the method 1400 shown in FIG. 14 may be performed by at least one processor included in the information processing system. For convenience of explanation, it will be described that each step shown in FIG. 14 is performed by a processor included in the information processing system.

The processor may generate a plurality of model sets (S1410). According to one embodiment, each of the plurality of model sets may be different in at least one of a data set used for learning, a learning method, a character set, or an entity set.

Afterwards, the processor may determine a target model set to be distributed among the plurality of generated model sets (S1420).

According to one embodiment, the processor may perform a test on each of a plurality of model sets and determine a target model set from among the plurality of model sets based on the performed test results. According to some embodiments, the processor may determine a target model set from among target model sets generated based on the document type.

Subsequently, the processor may distribute the determined target model set to the target system (S1430). Here, each of the plurality of model sets may include at least one machine learning model configured to recognize characters from document images. According to one embodiment, the processor may allocate computing resources for a target model set, and the target system may be configured to operate at least one machine learning model included in the target model set using the allocated computing resources. there is.

According to one embodiment, each of the plurality of model sets includes a first machine learning model configured to detect and output a plurality of regions in which characters are expressed in a document image, and a plurality of regions from the plurality of regions output from the first machine learning model. It may include a second machine learning model configured to recognize and output characters. Additionally, each of the plurality of model sets includes a third machine learning model configured to parse a plurality of characters output from the second machine learning model based on the plurality of entities and output a plurality of target texts associated with the plurality of entities. It may further include.

Thereafter, the processor may monitor the status of the target model set until the plurality of document images are applied as the target model set and the plurality of recognition results are output from the target model set based on the plurality of document images. According to one embodiment, the processor may determine a monitoring period for the target model set and monitor the state of the target model set during the determined monitoring period.

Additionally, the processor may determine the ratio of document images input to the target model set. Accordingly, the number of document images corresponding to the determined ratio among all document images may be input into the target model set.

According to one embodiment, the target model set includes a first machine learning model and a second machine learning model, and the processor may monitor the state of the first machine learning model and the state of the second machine learning model.

According to one embodiment, the processor calculates recognition accuracy for each of the plurality of model sets based on data output from each of the plurality of model sets and correct answer data, and performs recognition accuracy for each of the calculated plurality of model sets. Accuracy can be output. Additionally, the processor may calculate a trust score for each of the plurality of model sets based on state data for each of the plurality of model sets, and output the calculated trust score for each of the plurality of model sets.

The processor may apply the document image as the target model set, obtain recognition result data based on the document image, and generate learning data based on the document image and recognition result data. Subsequently, the processor may apply the generated training data as a target model set to perform additional learning on at least one machine learning model included in the target model set. According to one embodiment, the target model set may include a machine learning model configured to detect a plurality of sub-areas in which characters are expressed from a document image. Additionally, the processor outputs a user interface for providing a plurality of sub-areas detected from the machine learning model, receives a user input for changing at least one of the plurality of sub-areas through the user interface, and then based on the user input Thus, learning data with the changed sub-area determined as the correct answer data can be generated.

In some embodiments, the target model set may include a machine learning model configured to recognize a plurality of characters from a document image. Additionally, the processor outputs a user interface for providing a plurality of characters recognized from the machine learning model, receives a user input for changing at least one of the plurality of characters through the user interface, and then changes the user input based on the user input. It is possible to generate learning data where characters are determined as correct answer data.

In some embodiments, the target model set may include a machine learning model configured to recognize a plurality of target texts from document images. In addition, the processor outputs a user interface for providing a plurality of target texts recognized from the machine learning model, receives a user input for changing at least one of the plurality of target texts through the user interface, and then based on the user input Thus, learning data determined with the changed target text as the correct answer data can be generated. According to some embodiments, a document image may be output in a first area of the user interface, and a plurality of target texts related to a plurality of entities may be output in a second area of the user interface. When a specific target text is selected from among a plurality of target texts, at least one character associated with the selected specific target text may be visualized through a specific graphic element in the first area.

The above flowchart and above description are only examples and may be implemented differently in some embodiments. For example, in some embodiments, the order of each step may be changed, some steps may be performed repeatedly, some steps may be omitted, or some steps may be added.

The above-described method may be provided as a computer program stored in a computer-readable recording medium for execution on a computer. Media may be used to continuously store executable programs on a computer, or may be temporarily stored for execution or download. Additionally, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and There may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc.

The methods, operations, or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchange of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with this disclosure may be general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or It may be implemented or performed as any combination of those designed to perform the functions described in. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

For firmware and/or software implementations, techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and PROM ( on computer-readable media such as programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or marked data storage devices, etc. It may also be implemented as stored instructions. Instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

When implemented in software, the techniques described above may be stored on or transmitted through a computer-readable medium as one or more instructions or code. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a computer. By way of non-limiting example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or the desired program code in the form of instructions or data structures. It can be used to transfer or store data and can include any other media that can be accessed by a computer. Any connection is also properly termed a computer-readable medium.

For example, if the Software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, , fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. As used herein, disk and disk include CD, laser disk, optical disk, digital versatile disc (DVD), floppy disk, and Blu-ray disk, where disks are usually magnetic. It reproduces data optically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known. An exemplary storage medium may be coupled to the processor such that the processor may read information from or write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and storage media may reside within an ASIC. ASIC may exist within the user terminal. Alternatively, the processor and storage medium may exist as separate components in the user terminal.

Although the above-described embodiments have been described as utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the disclosure is not limited thereto and may also be implemented in conjunction with any computing environment, such as a network or distributed computing environment. . Furthermore, aspects of the subject matter of this disclosure may be implemented in multiple processing chips or devices, and storage may be similarly effected across the multiple devices. These devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in relation to some embodiments in this specification, various modifications and changes may be made without departing from the scope of the present disclosure as can be understood by those skilled in the art to which the invention pertains. Additionally, such modifications and changes should be considered to fall within the scope of the claims appended hereto.

210: communication terminal
220: network
230: Information processing system

Claims (20)

A method for distributing a model set associated with character recognition, performed by at least one processor, comprising:
generating a plurality of model sets;
determining a target model set to be distributed among the plurality of model sets generated; and
Distributing the determined target model set to the target system
A method for distributing a model set, comprising: each of the plurality of model sets including at least one machine learning model configured to recognize characters from document images.
According to paragraph 1,
Each of the plurality of model sets includes a first machine learning model configured to detect and output a plurality of regions in which characters are expressed in the document image, and a plurality of characters recognized from the plurality of regions output from the first machine learning model. A method for distributing a model set, including a second machine learning model configured to be output.
According to paragraph 2,
Each of the plurality of model sets is configured to parse a plurality of characters output from the second machine learning model based on a plurality of entities and output a plurality of target texts associated with the plurality of entities. A method for distributing a model set, further comprising models.
According to paragraph 1,
The step of distributing the target model set to the target system is,
Allocating computing resources for the target model set
Including,
The target system is configured to operate at least one machine learning model included in the target model set using the allocated computing resources.
According to paragraph 1,
After distributing the target model set to the target system,
Monitoring the state of the target model set until a plurality of document images are applied to the target model set and a plurality of recognition results are output from the target model set based on the plurality of document images.
A model set distribution method further comprising:
According to clause 5,
The step of distributing the target model set to the target system is,
Determining a monitoring period for the target model set
Including,
The step of monitoring the status of the target model set is,
Monitoring the state of the target model set during the determined monitoring period.
A method for distributing a model set, including:
In number 5,
The step of distributing the target model set to the target system is,
Determining the ratio of document images input to the target model set
Including,
A model set distribution method in which a number of document images corresponding to the determined ratio among all document images are input into the target model set.
According to clause 5,
The target model set includes a first machine learning model and a second machine learning model,
The step of monitoring the status of the target model set is,
Monitoring the state of the first machine learning model and the state of the second machine learning model
A method for distributing a model set, including:
According to paragraph 1,
Each of the plurality of model sets is different in at least one of a data set used for learning, a learning method, a character set, or an entity set.
According to paragraph 1,
calculating recognition accuracy for each of the plurality of model sets based on data output from each of the plurality of model sets and correct answer data; and
Outputting recognition accuracy for each of the calculated plurality of model sets.
A model set distribution method further comprising:
According to paragraph 1,
calculating a confidence score for each of the plurality of model sets, based on state data for each of the plurality of model sets; and
Outputting a trust score for each of the calculated plurality of model sets.
A model set distribution method further comprising:
According to paragraph 1,
The step of determining the target model set is,
performing a test on each of the plurality of model sets; and
Based on the performed test, determining the target model set from the plurality of model sets.
A method for distributing a model set, including:
According to paragraph 1,
After determining the target model set,
applying a document image to the target model set, and obtaining recognition result data based on the document image;
generating learning data based on the document image and the recognition result data; and
Applying the generated learning data to the target model set to perform additional learning on at least one machine learning model included in the target model set.
A model set distribution method further comprising:
According to clause 13,
The target model set includes a machine learning model configured to detect a plurality of sub-areas in which characters are expressed from the document image,
The step of generating the learning data is,
outputting a user interface for providing a plurality of sub-areas detected from the machine learning model;
Receiving a user input for changing at least one of the plurality of sub-areas through the user interface; and
Generating learning data that determines the changed sub-area as correct answer data based on the user input.
A method for distributing a model set, including:
According to clause 13,
The target model set includes a machine learning model configured to recognize a plurality of characters from the document image,
The step of generating the learning data is,
Outputting a user interface for providing a plurality of characters recognized from the machine learning model;
Receiving a user input for changing at least one of the plurality of characters through the user interface; and
Generating learning data that determines the changed characters as correct answer data based on the user input.
A method for distributing a model set, including:
According to clause 13,
The target model set includes a machine learning model configured to recognize a plurality of target texts from the document image,
The step of generating the learning data is,
Outputting a user interface for providing a plurality of target texts recognized from the machine learning model;
Receiving a user input for changing at least one of the plurality of target texts through the user interface; and
Generating learning data that determines the changed target text as correct answer data based on the user input.
A method for distributing a model set, including:
According to clause 16,
The document image is output in a first area of the user interface, and a plurality of target texts related to a plurality of entities are output in a second area of the user interface,
When a specific target text is selected from among the plurality of target texts, at least one character associated with the selected specific target text is visualized through a specific graphic element in the first area.
According to paragraph 1,
The step of determining the target model set is,
Determining a target model set from the generated target model set based on the document type
A method for distributing a model set, including:
A computer program stored in a computer-readable recording medium for executing the method according to any one of claims 1 to 18 on a computer.
As an information processing system,
Memory; and
At least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory
Including,
The at least one program is,
Create multiple model sets,
Determine a target model set to be distributed among the plurality of model sets generated,
Contains instructions for distributing the determined target model set to the target system,
An information processing system, wherein each of the plurality of model sets includes at least one machine learning model configured to recognize characters from document images.

Images